Analyzing log files can be a very time consuming process and it doesn’t seem to get any easier. In the past 12 years I have been on both sides of the table. I have analyzed terabytes of logs and I have written a lot of code that generates logs. When I started writing Loggly’s middleware, I thought it was going to be really easy and fun to finally write the perfect application logs. Guess what, I was wrong. Although I have seen pretty much any log format out there, I had the hardest time coming up with a decent log format for ourselves. What’s a good log format anyways? The short answer is: “One that enables analytics or actions.”
I was sufficiently motivated to come up with a good log format that I decided to write a paper about application logging guidelines. The paper has two main parts: Logging Guidelines and a reference architecture for a cloud service. In the first part I am covering the questions of when to log, what to log, and how to log. It’s not as easy as you might think. The most important thing to constantly keep in mind is the use of the logs. Especially for the question on what to log you need to keep the log consumer in mind. Are the logs consumed by a human? Are they consumed by a log management tool? What are the people looking at the logs trying to do? Debugging the application? Monitoring performance? Detecting security violations? Depending on the answers to these questions, you might change the places in your code that you emit log records. (Or even better you log in all places and add a use-case indicator as a field to your logs.)
The paper is a starting point and not a definite guide. I would expect readers to challenge it and come up with improvements and refinements of use-cases and also the exact contents of the log records. I’d love to hear from practitioners and get a dialog going.
As a side note: CEE, the Common Event Expression standard, covers parts of what I am talking about in the paper. However, the paper’s focus is mainly on defining guidelines for application developers; establishing a baseline of when log entries should be recorded and what information should be included.
Resources: Cloud Application Logging for Forensics – Paper – Presentation
Raffy,
I’m an application security engineer and software engineer and enjoyed your clearly written “Application Logging Guidelines”. I’ve spent the last several years working on the security-relevant layer of application logging and not very many people speak to the topic of what events to log and when to log them. I agreed with much of what you’ve written, but there was one issue I think needs clarification and one issue I seriously disagree with.
1. Application Layer Security Logging: Often web apps don’t have visibility of login, logout and other such events as their authentication front-ends are opaque. However initial session binding is often available and other issues such as new role association and other role management events are often available.
2. Maybe I misunderstood one of your points, but I believe I read that you recommend logging the “return” from subroutines/methods as ideal. While this would give you highly-granular log state it would also kill performance as well as consume unlimited storage and network bandwidth. Perhaps you meant that these return points should be instrumented, but not necessarily enabled? That would still have a significant impact on performance, but log forensics would be aided by highly selectable forensic capabilities.
Anyway, good work and I’m keeping an eye on CEE. BTW: Will CEE JSON or XML format be supported by Loggly? Their docs suggest something similar to CEE, but in a CEF-like “extension” format.
thanks,
eris
Comment by eris — September 21, 2011 @ 10:05 am
Raffy,
One other perennial sore spot regarding logging “requirements” in various regulatory schemes is the whole subject of log “review”. In the regulator’s fantasy worlds, we would give application subject matter experts (read ‘users with privileges’) access to the holy land of our security logs so they can examine them for “anything unusual”.
Of course, “anything unusual” is a very vague description and if an SME is forced to review the logs, that person will eventually page through the events so that the system will log his review as having happened. The end result is much wasted hours and security theater, but no real security.
How do you solve this problem?
Comment by eris — September 21, 2011 @ 12:15 pm
Eris,
Totally agree with both of your points in your first comment. The only problem with number two is that if you don’t enable them, you don’t have the data later for analysis. It might make sense to have a smart decision method to determine whether to log the return or not. It also depends on how modular your software really is.
Re: Loggly and CEE JSON. Not sure. I sure hope so though. You might want to post something in the forums (forum.loggly.com)
Not 100% sure what you mean by your log review comment. Sure, the paging through logs is not very efficient. That’s why over the past years I have been focusing on security visualization (http://secviz.org). That is a much more efficient and effective way of looking at your logs. And if you do it right, it can actually help you find interesting things.
Thanks for your comments!
Comment by Raffy — September 23, 2011 @ 2:53 pm