I launched a new portal that deals with visualization of log files:

http://secviz.org

The portal can only survive if people- you – take an active part in contributing content.

There are multiple resources available where community input is most welcome:

* Graph Exchange: The idea is that people can submit their graphs, explain why they think the graphs are useful, and how they generated them.
* Parser Exchange: To generate graphs, you need to parse your data. This is a place where you can submit your parsers.
* Links: A whole bunch of links around data analysis and visualization.
* Discussions: A free forum where you can start discussions around the topics of log visualization and analysis.

Let me know what you think and most importantly, submit your graphs

I keep running into these little annoyances in Linux. (And as I said here before, I love Linux, but there are some things which are just bad.) This time I was trying to see what happens if you lock an accound. You didn’t even know you could do that?

passwd -l 

Do you know what syslog has to say about this?

Nov 14 16:35:12 zurich passwd[21226]: password for `test' changed by `root'

And even worse, if you unlock:

passwd -u 

Linux says:

Nov 14 16:35:12 zurich passwd[21226]: password for `test' changed by `root'

Great! What am I supposed to do with this? Is a password change really the same as a lock out of a user?

To continue on the path of auditing and such, have you tried to configure an automatic lock-out after a certain amount of failed logins? Good luck. After a while you might find pam_telly. You have to use this PAM module to achive that lockout. You can configure after how many failed passwords an account gets locked. Again, why is this in such a hidden module? Why not built-in? Is anyone going to rebuild the authentication sub-system? Please? And if you are at it, rethink the whole logging infrastructure too! Don’t forget to use a common log format, a specific fixed format that enforces certain information and is parsable! Stop logging copyright messages into syslog (Ok: dhclient?).

There is all this talk about event interoperability standards or logging standards. Don’t we have enough of them? IDMEF, IDXP, SDEE, WELF, CBE, RDEP, OPSEC. All of them are approaches to solve the same problem: Simplify or enable the interoperability of devices and applications. Does anyone support these standards? No! The question is why? Here is my answer:
Have you ever looked at these standards? Noticed anything? These guys are all trying to solve many problems at once. I already blogged about the four different types of log standards that we need. One important things it that the transport needs to be separated from the format! SDEE for example requires SOAP as a transport. Have you implemented SOAP messaging ever? What an effort. I don’t want to do it in my applications. I want something easy! Why not using simple transports? What about files or syslog. And when I say syslog, I don’t mean the gibberish you can log in the message, but I mean the transport. Very simple! Very easy to implement!
Some standards are using XML. It’s just too much work to implement XML messages. You need to keep track of the elements, the hierarchy, the attributes, validate against the DTD, the Schema, etc. And you need a transport that can support it. Nevertheless, there are a few advantages to XML: You can express lists and you can enforce a very well defined format. But that’s it.
So my point being, use a text-based format. Do we have any standards in that arena? Well, there is CEF (Common Event Format). And that’s it. I don’t konw of any others. The standard is very well designed. And not by academics or people that have never seen a log file before, but by people that have seen hundreds of different log formats. A log standard needs some other considerations. Things like event IDs or severities. Things that an event consumer is interested it! But that’s a topic for another entry.
There is a second point that you can make agains text-based formats (the first point being that lists are hard to express), which is speed. I completely agree, if you want speed, you need to go binary! Period. Use NetFlow as an example where you send some kind of a template first and then you send the messages in that format. However, there are other drawbacks: it’s harder to implement (you need preprocessing), not every transport is suited for it, etc.
So to conclude: We really need three logging standards:

The problem of how you call something is not something I think about conciously very often, but it became really obvious to me that it is important to name things and define what they really mean every so often. In my daily work I use the words Event and Log Entry all the time. While talking to developers and other geeks, it has never been a problem, but I was talking to some other groups lately, outside of my company and when I mentioned the word event it took me a while to understand that they did not think about an event the way I did. An event for them was an incident, a physical event, the constellation of things coming together and causing something to happen. For me an event is something I use very loosely. An event gets generated by a device. It’s the same as a log entry. It’s a “string” that describes what happened. Windows for example generates events. They get collected in the event log. But again, I am using the term very loosely. What’s a log entry then in contrast to an event? Hmm… And is a tcpdump record a log entry or an event or what is it? Hard to say. I guess it takes the effort of someone to define all that. I might…

I am scribbling on another book chapter. This time it’s for a visualization book. I am writing about how to analyze firewall and IDS logs. I am using line graphs and treemaps to do so. Guess what tool I am using to generate all the graphs. Yes. AfterGlow.
I am not quite done with writing, but am pretty happy with the way it shapes out. The chapter is not going to be highly technical. I am not going into how to configure AfterGlow and parse log messages and such. I focus more on the process-level. It is quite an interesting experience to put something into words that you intuitively do all the time.
I am not sure when the book is actually going to come out, but I will post here when it’s available.

There is a lot of talk around interoperability standards lately.Following these discussions, it seems to me that people are intermixing a lot of different topics:

a) Log format (syntax)
b) Event transport
c) Event classification (also called taxonomy, categorization, grammar)
d) Logging recommendations (what events specific devices should report AND what fields they should contain as a minimum

I would really like to see future discussions broken up into these four groups!

Quite a while ago, the goal of loganalysis.org was to collect log files of all kinds of devices to build up a repository for the community. Unfortunately that effort has not been too successful. I just stumbled accross a new effort driven by splunk:
www.splunk.com/base. There are quite a few syslog messages on there already. What i don’t like is that most of the messages are some kind of exceptions of some java applications. I don’t really care about those things. Well. Hopefully there are going to be more people adding logs…

A horrible acronym. I know. We had a working session during the RSA conference to talk about XCCDF-P. For those not familiar with XCCDF, it has to do with policy definitions and uses OVAL to implement the checks.

XCCDF-P (which will hopefully get renamed pretty soon to something else, and hopefully not to CPN (Common Platform Names) [We already have CVE, CME, and CCE]) is an effort to standardize platform names. What’s the problem? Well, if I have two scanners analyzing a system of mine, one of them might report that I am running a “Windows 2000”, the other one might say “Win2K”. This is really the same, but how would a machine know? That’s where the standard is trying to clean things up. You wouldn’t belive how much discussion this topic actually involves. We met for about an hour and had plenty of things to discuss, not even closely getting to an agreed-upon solution. However, the problem is defined and we all agreed upon the the necessity to solve the problem! Stay put for an update soon and hopefully a quick turn around with a solution draft.

Wow. I just found this pretty awesome TreeMap tool. The data format it reads is pretty easy and I quickly built a file with some of my firewall data. Well, fake firewall data 😉

What you see in here is first the color: green are firewall passes, red are blocks, then the hierarchy is such that the target system is top, then the target IP and then you see the date inside of the boxes, when the access happened.

Well, the tool is pretty awesome. Lots of interactivity. You define the hierarchies manually, on the fly it updates the graph. Then you can color and filter and all kinds of nifty things. Try it out.

I am still sitting in the airplane and the next article from the ISSA Journal from November 2005 that catches my attention is the “Log Data Management: A Smarter Approach to Managing Risk”. I have only a few comments about this article:

1. The author demands that all the log data is archived, and archived unfiltered. Well, here is a question: What is the difference between not logging something and logging it, but later filtering it out? What does that mean for litigation quality logs?
2. On the same topic of litigation quality data, the author suggest that a copy of the logs are save in the original, raw format while analysis is done on the other copy. I don’t agree with this. I know, in this matter my opinion does not really count and nobody is really interested in it, but I will have some proof soon that this is not required. I am not a lawyer, so I will not even try to explain the rational behind allowing the processing of the original logs and still maintaining litigation quality data.
3. “Any log management solution should be completely automated.” While I agree with this, I would emphasize the word should. What does that mean anyways? Completely automated in the real of log management? Does that mean the log is archived automatically? Does it mean that the log management solution takes action and block systems (like an IPS)? There will always need to be human interaction. You can automate a lot of things, including the generation of trouble tickets, but at least then, an operator will be involved.
4. Why does the author demand that “companies should look for an applicance-based solution”. Why is that important? The author does not give any rational for that. I can see some benefits, but there are tons of draw-backs to that approach too. I yet have to see a compelling reason why an appliance is better than a custom install on company approved hardware.
5. In the section about alerting and report capabilities, the author mentiones “text-based alerts”, meaning that rules can be setup to trigger on text-srings in log messages. That’s certainly nice, but sorry, it does not scale. Assume I want to setup a trigger on firewall block events. I can define a text-string of “block” to trigger upon. But all the firewalls which call this not a block, but a “Deny” will not be caught. Have you heard of categorization or an event taxonomy? That’s what is really needed!
6. “… fast text-based searches can accelerate problem resolution …” Okay. Interesting. I disagree. I would argue that vsualization is the key here. But I am completely biased on that one 😉
7. Another interesting point is that the author suggest that “… a copy [of the data] can be used for analysis”. Sure. Why not, but why? If the argument is litigation quality data again, why would compression, which is mentioned in the next sentence be considered a “non-altering” way of processing the data. If that is the argument. I would argue that I can work with the log data by normalizing it and even enriching the data without altering it.