There is all this talk about event interoperability standards or logging standards. Don’t we have enough of them? IDMEF, IDXP, SDEE, WELF, CBE, RDEP, OPSEC. All of them are approaches to solve the same problem: Simplify or enable the interoperability of devices and applications. Does anyone support these standards? No! The question is why? Here is my answer:
Have you ever looked at these standards? Noticed anything? These guys are all trying to solve many problems at once. I already blogged about the four different types of log standards that we need. One important things it that the transport needs to be separated from the format! SDEE for example requires SOAP as a transport. Have you implemented SOAP messaging ever? What an effort. I don’t want to do it in my applications. I want something easy! Why not using simple transports? What about files or syslog. And when I say syslog, I don’t mean the gibberish you can log in the message, but I mean the transport. Very simple! Very easy to implement!
Some standards are using XML. It’s just too much work to implement XML messages. You need to keep track of the elements, the hierarchy, the attributes, validate against the DTD, the Schema, etc. And you need a transport that can support it. Nevertheless, there are a few advantages to XML: You can express lists and you can enforce a very well defined format. But that’s it.
So my point being, use a text-based format. Do we have any standards in that arena? Well, there is CEF (Common Event Format). And that’s it. I don’t konw of any others. The standard is very well designed. And not by academics or people that have never seen a log file before, but by people that have seen hundreds of different log formats. A log standard needs some other considerations. Things like event IDs or severities. Things that an event consumer is interested it! But that’s a topic for another entry.
There is a second point that you can make agains text-based formats (the first point being that lists are hard to express), which is speed. I completely agree, if you want speed, you need to go binary! Period. Use NetFlow as an example where you send some kind of a template first and then you send the messages in that format. However, there are other drawbacks: it’s harder to implement (you need preprocessing), not every transport is suited for it, etc.
So to conclude: We really need three logging standards: