I am really late to the game. But finally I read draft-ietf-syslog-protocol-23. This is the new draft for revising the syslog protocol.

Here are some of my comments that I also submitted officially:

• Let me say this first: I really like some of the changes that have been incorporated.
• Syslog message facility: Why still keeping this? The only reason that I see people using the facility is to filter messages. There are better ways to do that. Some of the pre-assigned groups are fairly arbitrary and not even really implemented in most OSs. UUCP subsystem? Who is still using that? I guess the reason for keeping it is backwards compatibility? If possible, I would really like this to be gone.
• Priority calculation: The whole priority field is funky. The priority does not really have any meaning. The order does not imply importance. Why having this at all?
• Timestamps: What’s the reason for having the “T” in the timestamp? Having looked at hundreds of different log formats, I have never seen anything like that. Why doing this?
• Hostname: I am not comfortable with the whole hostname spec. I like that there is an ordering and people are supposed to use FQDNs, but there are many questions about this. To start with, in a lot of UNIX configurations, /etc/hosts contains an entry like
  127.0.0.1   localhost.localdomain  localhost
  The second column is the FQDN (technically). Is that one that can be used? Can you make it clear that this is not what should be used? Same for 127.0.0.1 or the loopback address in general. How does a machine know whether an IP address is static or dynamic? How does a logging application know? I don’t think you will ever know that. Did you mean a private versus a public address? That might be interesting. Furthermore, it should specify which interface’s IP address to use. The interface that the message is sent out on?
• Under the section of PROCID: The text is imprecise. This number is not the process ID of the syslog process, it’s the ID of the writing process. The third paragraph talks about detecting restarted applications and somehow mixes in the syslog process. (“might be assigned the same process ID as the previous syslog process“.) This is not clear at all and very very confusing.
• MSGID: Make clear that this ID is local to the application. It’s not a global ID at all.

The biggest issue I have around the SD-ID field:

• I like that the user can extend the set of registered IDs.
• Why is this structure so complicated? Why not going with a simple set of key-value pairs? This whole structure thing is so complicated. Parsing it, you need to keep state! You need to remember the SD-ID for each SD-PARAM. Why introducing this? Just stick with simple key-value pairs. That makes parsing easier. Much easier. And it makes the events easier to produce as well.
• By keeping an explicit message field (the unstructured part), you encourage people to still log in that way. I recommend using an explicit field (or parameter) that can be used to include human readable text. Instead of this:
  <165>1 2003-08-24T05:14:15.000003-07:00 192.0.2.1 myproc 8710 - - %% It's time to make the do-nuts.
  use:
  <165>1 2003-08-24T05:14:15.000003-07:00 192.0.2.1 myproc 8710 - message="%% It's time to make the do-nuts."
  or really:
  1 2003-08-24 05:14:15.000003-07:00 host=192.0.2.1 process=myproc procid=8710 message="%% It's time to make the do-nuts."
• I definitely like the consideration of some of the special fields (structured data IDs). However, they should be used as simple keys (or parameters) that have special meaning.
• Parameter – origin: What does it mean to have multiple origins IPs? Is that a syslog forwarding chain? The document does not say anything about that. Also, we already have the host field in the beginning of the syslog messages. What’s the relationship to that? Or is origin something completely different?
• Parameters – I would really like to see some use-cases for all of the IDs. Especially the sequenceId. I am assuming this is something that the syslog daemon assigns, not the logging application. Right? I think that needs to be clearer. For the sequenceId, what happens for forwarded messages? Are these IDs local? Are they forwarded along with a message? Also, how does the logging application know about the timeQuality? Or if that something that the syslog daemon assigns, how does it know?
• I would really like to see the parameters to go away and have a generic key-value extension. In addition, IANA should have a set of allowed/defined keys. The parameters should be part of those. Each key has a special meaning (semantics). There should be a whole lot of them: src_ip, user_name, etc. Each producer should be free to add additional keys, realizing that not all consumers would understand their semantics. However, the consumers could still read them.

That’s it for now… Let’s see what some of the reactions are going to be.

[tags]IANA, syslog, syslog protocol, IETF, logging[/tags]

VizSec is a fairly academic conference that brings together the fields of security and visualization. The conference had an interesting mix of attendees: 50% came from industry, 30% from academia, and 20% from government. I had the pleasure of being invited to give a talk about DAVIX and also participate on a panel about the state of security visualization in the market place.
The highlight of the conference was definitely Ben Shneiderman’s keynote. I was very pleased with some of the comments that Ben made about the visualization community. First he criticized the same thing that I call the “industry – academia dichotomy”. In his words:

“[There is a] lack of applicability of research.”

I completely agree and if you have seen me talk about the dichotomy, I outline a number of examples where this becomes very obvious.
The second quote from Ben that I would like to capture is the following:

“The purpose of viz is insight, not pictures”

Visualization is about how to present data. I am not always sure that people understand that.
Unfortunately, I wasn’t prepared to capture what Ben said about my book (Applied Security Visualization.) He brought his copy that I had sent him. He talked about the book for quite a bit and specifically mentioned all the treemaps that I have used to visualize a number of use cases. I felt very honored that Ben actually looked at the book and had such great things to say about it. The following lunch with Ben was a great pleasure as well, filled with some really interesting visualization discussions.

I recorded a couple of podcasts and did some interviews lately about the book. If you are interested in listening in on some of the press coverage:

• Security Wire Weekly: Security Visualization, Interview with Robert Westervelt from TechTarget.
• Networking data visualization not just for pointy-headed bosses, Interview with Michael Morisy

More information about the Applied Security Visualization book is on the official book page. I am working on figuring out where to put an Errata. There were some minor issues and typos that people reported. If you find anything wrong or you have any generic comments, please let me know!

I just saw the first Amazon review for my book. I just don’t understand why the person only gave it four stars, instead of five 😉 Just kidding. Thanks for the review! Keep them coming!

The Applied Security Visualization book is DONE and available in your favorite store!

Last Tuesday when I arrived at BlackHat, I walked straight up to the book store. And there it was! I held it in my hands for the first time. I have to say, it was a really emotional moment. Seeing the product of 1.5 years of work was just amazing. I am really happy with how the book turned out. The color insert in the middle is a real eye-catcher for people flipping through the book and it greatly helps making some of the graphs better interpretable.

I had a few copies to give away during BlackHat and DefCon. I am glad I was able to give copies to some people that have contributed by inspiring me, challenging me, or even giving me very specific use-cases that I collected in the book. Thanks everyone again! I really appreciate all your help.

People keep asking me what the next project is now that the book is out. Well, I am still busy. secviz.org is one of my projects. I am trying to get more people involved in the discussions and get more people to contribute graphs. Another project I am starting is to build out a training around the book, which I want to teach at security conferences. I have a few leads already for that. Drop me a note if you would be interested in taking such a training. Maybe I will also get some time to work on AfterGlow some more. I have a lot of ideas on that end…

During DefCon, I recorded a PodCast with Martin McKeay where I talk a little bit about the book.

I just finished reading the NIST 800-41 draft about “Guidelines on Firewalls and Firewall Policy“. The guideline does a great job of outlining the different types of firewalls that exist and how to correctly setup a firewall architecture.

The topic that falls fairly short is logging:

• Section 5.2.3 (Configuring Logging and Alerts) mentions logging very briefly.
• I am positively surprised that it mentions the logging of rule changes on the firewall, which is inherently hard in, for example, IPTables.
• NIST asks for storing the logs locally on the firewall. I don’t agree with that at all. I don’t care whether the logs are kept locally. What I really care about is that the logs are centrally collected. Or in a very small environment, that there are logs at all.
• I was really hoping that this was finally the document which would outline what to log exactly. What traffic should I be logging on the firewall? All the traffic? Just denied packets? Do you log on the incoming interface? And so on. None of these questions is addressed. Not even whether passed traffic should be logged at all. There should at least be some discussion around that.
• Log analysis is not mentioned either. I was hoping that aside from just logging recommendations, the guideline would quickly mention what to do with the log files. How do you use them? Are they meant mainly for forensic purposes or are they used for proactive analysis? This would help justify the storage cost of the logs and push some implementations to actually implement logging.

I sent this blog post to the authors of the guidelines. Hopefully they are going to address some of this. And again, the general structure and contents of NIST 800-41 are great!

I am presenting at the FIRST 2008 conference in Vancouver next week. I am speaking on my birthday, June 25th, from 9.50 until 12.50. The topic is “Applied Security Visualization” – the same as my book title. I am going through some of the material from the book and show how visualization can be used to analyze log files.

Some of the highlights:

• I am going to show how you can use Splunk to manage not just single-line logs, but also analyze multi-line data, such as data from top, ps, etc.
• I am showing how you can use AfterGlow with Splunk.
• I am probably going to show a sneak peak of DAVIX. The Data Visualization and Analysis Linux (DAVIX) is a live CD that will be released at BlackHat this year.

The Applied Security Visualization book is slowly coming together. I finished implementing all the reviews from my reviewers yesterday. This means I am almost done. The only thing left is the introduction.

By the way, my reviewers were absolutely amazing. I couldn’t have wished for a better team. Thanks guys!

The rought-cuts version of “Applied Security Visualization” is now also available. It’s an electronic version of, I think, 4 of the chapters. You can also pre-order the book on Amazon. This is all really exciting. Finally, after 1.5 years, the book is close to be done. Let’s hope for a launch in August, at BlackHat!

I just joined the program committee for this year’s WASL conference. I am really curious what papers will be submitted for this. Talking about papers, I have been busy lately reviewing papers for RAID and soon the papers for VizSec are due as well. While I enjoy reading these papers, it’s been too busy lately with finishing my book and looking these papers. But at least the book is getting close!

Here is the description for WASL:

Join us in San Diego, CA, December 7, 2008, for the First USENIX Workshop on the Analysis of System Logs. System logs represent a rich source of information for the analysis and diagnosis of system problems and prediction of future system events. However, their lack of organization and the general lack of semantic consistency among the information from various software and hardware vendors means that most of this information content is wasted. WASL ’08 will focus on novel techniques for extracting more information from existing logs and on methods to improve the information content of future logs.

This week, the RSA Security conference is taking place in San Francisco. Just a few things I want to capture:

• I will be speaking on Thursday morning 8am! Topic of the presentation: “A Journey Through Security Visualization“. I am co-presenting with Alain Mayer from RedSeal.
• I am announcing the rough-cuts version of my Applied Security Visualization book. This is an online version of three chapters of the book. You can get an electronic version and give me feedback on them. Not that I didn’t have enough to do with the comments from my reviewers already 😉
• I am going to be hanging out at various events and parties during the week. My schedule during the day is really full at this point, but I would love to meet you during the evening activities. Hit me up or check my Twitter feed for where I am at: @zrlram.
• I am most likely going to be at the Blogger Meetup on Wednesday. [If Splunk is not going to drag me into the Analyst dinner.]

So, given all this, it’s going to be both a busy and a fun week. Hope to see ya all there!

[tags]visualization, rsa security, visualization[/tags]