December 7, 2005

Commen Event Format / Standard

Category: Log Analysis — Raffael Marty @ 12:51 pm

There is an interesting thread on the log-analysis mailinglist about regex-less parsing of messages. The problem is a very old one. Every device out there is logging in some strange way, making it incredibly time-consuming for event consumers (such as ArcSight), to parse the messages and normalize them.
There have been attempts to standardize events, such as IDMEF, which tried to tackle IDS messages. It’s kind of sad, but there is not a single IDS that I know of, which really uses this event exchange format. A lot of IDSs support it, but it’s not their main transport. Then there are tons of other attempts from BEEP to RDEP to SDEE and alike. They are all nice, but guys, we need something that is

  • easy to implement,
  • scales to high event rates,
  • is extensible to support not just security devices (for sure not just IDSs),
  • and is MACHINE READABLE (not human readable) [when are you people going to realize that logs are not read by humans anymore, but by machines?].
  • All the past attempts of standardizing event formats are not enough, now Microsoft comes out with yet another event logging format. I have to admit, I only quickly glanced over it, but it’s XML again. That’s just SLOW! Huge overhead!
    Also, why do people always define the transport when they are trying to standardize log messages? Leave the transport to the devices. They will figure that one out. In the worst case, people can just use syslog which is widely deployed and has it’s problems. But you know what? At least the burden of complying with the standard is incredibly low. Just send a syslog message. Even I can do that. If you asked me to implement BEEP, I don’t think I would even start thinking about complying with the standard…

    Sorry for the long post and rant, but this is just a bit frustrating …

    1 Comment »

    1. You missed one more criteria: “… and will be accepted”

      Comment by Anton Chuvakin — December 9, 2005 @ 8:02 pm

    RSS feed for comments on this post. | TrackBack URI

    Leave a comment

    XHTML ( You can use these tags): <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong> .