My R journey seems to be over… It’s just too complicated to get charts and plots right with R. I found a new library that I staerted playing with: chart director. I am using Perl to drive the graph generation and it’s actually fairly easy to use. There are some cumbersome pieces, such as seg-faults when you have NULL values in a matrix that you are drawing, but that’s things I can deal with ;

Here is a sample graph I generated: (I know, the color selection is fairly horrible!)

[tags]R, charts, visualization, perl[/tags]

Log analysis has shifted fairly significantly in the last couple of years. It is not about reporting on log records (e.g., Web statistics or user logins) anymore. It is all about pinpointing who is responsible for certain actions/activities. The problem is that the log files do oftentimes not communicate that. There are instances of logs (mainly from network centric devices), which contain IP addresses that are used to identify the subject. In other instances, there is no subject that can be identified in the log files at all (database transactions for example).

What I really want to identify is a person. I want to know who is to blame for deleting a file. The log files have not evolved to a point where they would contain the user information. It generally does not help much to know what machine the user came from when he deleted the file.
This all is old news and you probably are living with these limitations. But here is what I was wondering about: Why has nobody built a tool or started an open source project which looks at network traffic to extract user to machine mappings? It’s not _that_ hard. For example SMB traffic contains plain-text usernames, shares, originating machines, etc. You should be able to compile session tables from this. I need this information. Anyone? There is so much information you could extract from network traffic (even from Kerberos!). Most of the protocols would give you a fair understanding of who is using what machine at what time and how.

[tags]identify correlation, user, log analysis, user mapping[/tags]

I was trying to get my Ubuntu desktop to use Beryl, just like my laptop does. Unforunately, my NVidia drivers didn’t quite want to do what I wanted them to do. Long story short, at some point I remembered to check in the log files to see whether I could determine what exactly the problem was. Where should I look first? /var/log/messages And right there it was:

May 11 11:15:12 zurich kernel: [ 2503.193111] NVRM: API mismatch: the client has the version 1.0-9631, but
May 11 11:15:12 zurich kernel: [ 2503.193114] NVRM: this kernel module has the version 1.0-9755. Please
May 11 11:15:12 zurich kernel: [ 2503.193115] NVRM: make sure that this kernel module and all NVIDIA driver
May 11 11:15:12 zurich kernel: [ 2503.193117] NVRM: components have the same version.

Beautiful. That’s exactly what I needed to know. But hang on a second. Isn’t this a syslog entry? Wow. It just hit me. While I really liked the verbose output, I was trying to think about how I would parse this thing. How would I normalize this message to later apply machine logic to further process this? Aweful!

I guess my conclusion would be that we need two types of Syslogs! One that logs machine readable log entries and one for humans. Is that really what we want? Maybe the even better solution would be to only have a machine readable log and then provide an application that can read the log and blow the contents up to make it readable for humans!

Where is CEE when you need it?

Although I work in the log/event management space and therefore help organizations to gather more information about people, I am a big opponent of personal information collection.

I flew back from Switzerland to San Francisco after my Christmas break and was in for a surprise. Not only did they want my passport (which I can sort of understand ;), but they also wanted me to fill out an additional form with my address in San Francisco, a contact person, etc. Why do they need all that? And then there is still the controversy about the airlines giving passenger information to the TSA and possibly other US agencies. I just don’t know what they use all this information for? To flag potentially dangerous passengers? What was the rate of false positives for that? I wish everyone had stringent laws as the EU for personal data. At least I would have a chance to find out what the data is that they have about me and possibly correct it!

Are you a non-US citizen, and if so, did you enter the US lately? Yes? Picture taken, finger prints (soon to be 10, not just 2). Even more data they collect. I’ve got to tell you, it’s not just the wait in the immigration hall that annoys me. It’s all the data they collect. And that’s what tirggered my post. I wouldn’t have that much of a problem, if they actually told me what they were going to do with the data and kept it safe.

Maybe they are starting to rethink the “data collection” after more and more of the US agencies are suffering data leaks. Now the TSA itself. Hopefully they realize that they should either start to be serious about data security or stop collecting information!

I have some more detilas on the CEE effort, which is captured in this CEE Brochure. The most interesting part is probably page two where the benefits are outlined. This effort will continue by tackling one of the four standard areas after the other. I have a feeling that we will tackle the taxonomy part first. I can already see it, this is going to be HARD!

As Anton mentioned, there is a new event logging standard in the works. What Anton did not mention is the four areas that you need to talk about when you talk about a logging standard. Well, here they are:

1. Common Event Syntax, like CEF
2. Common Event Taxonomy. This is where you attach “meaning” or “semantics” to an event. There are a few proprietary ones, nothing standardized though.
3. Common Event Transport
4. Common Event Representation, defining what a device should log. An operating system should log user logins for example.

And don’t mix these things. The transport has nothing to do with the syntax! I don’t want to implement a SOAP environment to transport some events. Unfortunately a few companies and even standards have made that mistake! I don’t want to mention anyone here…
Stay tuned for http://cee.mitre.org to go live and learn more about all of this.

I came accross this very well done Web Log Analysis. The author uses a 3D scatter plot to plot certain aspects of his Web server log. He uses gnuplot to do so. What I like in particular is his discussion of the output and the way he positions scatter plots to find correlated event fields.

I am finally biting the bullet. I will start to really anonymize my graphs. In order to do so, I was trying to find a tool on the Web which does that. Well, as you can probably imagine, there is non which does exactly what I wanted. So i wrote my own anonymization script. To safe you some hassle, also download the Anonymous.pm file.

This is how you use the script on a CSV file:

cat /tmp/log | ./anonymize.pl -c 1 -p user

Credits to  John Kristoff who wrote the Anonymous.pm module for Perl.

I read a lot of research papers and security articles. I am getting so tired seeing all these tools, research papers, and new algorithms that propose new approaches in computer security and then as a proof, they are solving one of the “old” problems: Detecting worms, portscans and finding peer-to peer traffic. Guys, it’s been done. We don’t need any more tools to do it. It’s easy and nothing to show off with!
Show me that other use-cases can be solved with your new approch. That will not only tell me that you actually thought about the problem space, but it will help the security community at large to tackle new problems (maybe some that they were not even aware of)!

It was a bad oversitght that secviz.org did not have an RSS Feed. But now there is one! The feed contains all new content posted to the portal, including comments. Subscribe so you don’t have to check back all the time to see whether there is new content.

[tag]security visualization[/tag]