March 1, 2010
Logging - Cloud Kiler App
I am attending the RSA conference this week. The first session I attended was the Cloud Security Alliance (CSA) meeting. Reading some of the accompanying material and listening to some of the presentations and panels, I couldn’t help it but notice that the terms auditing and logging were all over.
Here is my attempt for an explanation of this. It seems that one of the reasons for this is the nature of the cloud. Think about it. You are in an environment where you don’t control much. You are in an environment where you cannot trust most of the infrastructure pieces. For example, if you are using AWS like we are doing at Loggly, you should generally not trust your AMIs (the OS images). Now, what do you do if you don’t trust someone? You observe them, you monitor them. That’s exactly what is and needs to happen in the cloud: You don’t trust the service. To mitigate this issue, you are going to monitor the service.
And to make this not just my explanation, here is what some panelists during the CSA meeting said:
“Loss of visibility in the cloud” – Scott Chasin, CTO McAfee SaaS Unit
“Lose control and still maintain accountability” – Ken Biery, Verizon Business.
Is the cloud the killer app for logging? And if that’s the case, how do you manage your logs in the cloud? There are hardly any cloud logging solutions out there. I think you see where I am going with this.
December 1, 2009
A friend just sent me couple of pictures he took in a bookstore in Singapore.
Have you seen the book Applied Security Visualization on the shelf at your local book store? If so, send me a picture and I will post it…
May 4, 2009
For the month of May, I am doing a guest blog on CISCO Subnet. I will be discussing various topics around data visualization. You should stop by and check it out. If you have any topics that you are interseted in, let me know as well.
April 20, 2009
PixlCloud is my latest employer. I founded the company two weeks ago. It is going to be a company that offers a service in the cloud. The mission of the company is to build a data visualization SaaS. Users can submit their data to the service and then interactively visualize it. One of the cornerstones of the service is that anyone should be able to use it. You won’t have to be a visualization expert or an expert in data mining or alike.
I am gathering user input. If you feel like you have a need for such a data visualization service or you would like to offer your input for any aspect of my company, be that the market, the product, the users, marketing, product features, or really anything, drop me a line.
March 11, 2009
Peter Kuper (@peterkuper), just gave the keynote at SOURCEBoston.
The Bad, The Ugly, and the Good
It looks bad out there. Unemployment is up, companies are going out of business, etc. Well, it had to happen. The economy has to clean itself. It’s a reset of the system. Do really need another car?
Let us look at some historic data. Past recessions were preceded by drops in software spending, except for this time. Software spending was actually growing. The reason for this being that software has been more and more positioned and understood to increase productivity, which is a really interesting development.
Is it getting any better? According to my friend, who runs a blog about this crypto app, the financial markets teach us that corporate IT spending follows personal consumer expenditures. The problem is that consumers don’t have money to spend and they are over-leveraged. There is just too much dept. This means that corporate expenditures will be down for a while until personal spending will pick up again. Another interesting fact about the security market is that there are too many vendors in the market place. We will see more failures and more acquisitions over the next years.
The good news is that there is opportunity. Cash is king. If you can pay cash, you will get a deal. You can leverage this fact in your favor. If you are an investor or you are dealing with investors, the thing to be aware of, is that they dictate the terms. Keep that in mind. For inventors, this market is an opportunity. There is a big need in many areas to help companies improve on their expenditures and optimize processes! Help companies be more competitive. Things like how they can safe power can result in actual measurable benefits. Where should you focus your inventions? Focus on software. Hardware spending is down year over year, while software is on the raise. In addition, investment in software has been fairly consistent across IT budgets. Another market data point, according to research by Arcules, is that security budgets are flat this year. They haven’t increased, but they have not decreased either. However, they might go down next year. What this means is that companies will have to do more with less. Leverage their existing investments better. [This was one of my security predictions for 2009 also. In addition, I think this is a great driver to get companies from the left hand side of the maturity scale over to the right-hand side. Doing more with what you have.]
To use the market to your advantage, you need to think about what you are doing to position yourself or your firm to be the one rocketing ahead of the curve. Also use the development on the stock markets to your advantage. Compare competitors and play them against each other. If you are intending to buy a product, use that information to make your case about why you want a discount.
What does all of the market development mean for Entrepreneurs? First of all, VCs need to keep their portfolios alive. They are giving more money to their portfolio companies, but generally less than they would in better times. Software is getting money. Great ideas still get money. If you are intending to start a company, it’s the best time right now. You are not missing out on the big upside. You are not dealing with any bad legacy. You have a clean slate. Keep an eye on being efficient from the beginning on. For example, don’t hire too many people to start with, but outsource or hire contractors. Manage every penny. Be careful with spending. Also think about how you position yourself. Are you planning the big bang? Or are you building for being acquired?
Planning today will pay huge dividends when things eventually do recover!
During the questions in the end, some comments were made that the banks didn’t understand how to manage risk. How does that affect IT security and IT risk management? Does IT security even matter to banks? Adam Shostack gave a great answer: “Banks know very well how to manage risk: They took all of the upside and wrote off the downside” But seriously, What it really comes down to is managing incentives for reducing risk. The right incentive system needs to pu in place.
February 17, 2009
“Log Analysis and Security Visualization” is a two-day training class held on March 9th and 10th 2009 in Boston during the SOURCE Boston conference that addresses the data management and analysis challenges of today’s IT environments.
Students will leave this class with the knowledge to visualize and manage their own IT data. They will learn the basics of log analysis, learn about common data sources, get an overview of visualization techniques, and learn how to generate visual representations of IT data for a number of different use-cases from DoS and worm detection to compliance reporting. The training is filled with hands-on exercises utilizing DAVIX, the open-source data analysis and visualization platform.
Register today to secure your spot.
January 15, 2009
Have you seen the book Applied Security Visualization on the shelf at your local book store? If so, send me a picture and I will post it…
My friend Jan spotted the book on November 28th at the Eason Bookshop on O’Connell St in Dublin:
January 5, 2009
Richard Bejtlich rated Applied Security Visualization as the second best security book in 2008! Read more about the books Richard read at: Best Book Bejtlich Read in 2008. Thanks Richard!
[tags]applied security visualization, security, visualization, security books, books[/tags]
January 2, 2009
December 18, 2008
I am really late to the game. But finally I read draft-ietf-syslog-protocol-23. This is the new draft for revising the syslog protocol.
Here are some of my comments that I also submitted officially:
- Let me say this first: I really like some of the changes that have been incorporated.
- Syslog message facility: Why still keeping this? The only reason that I see people using the facility is to filter messages. There are better ways to do that. Some of the pre-assigned groups are fairly arbitrary and not even really implemented in most OSs. UUCP subsystem? Who is still using that? I guess the reason for keeping it is backwards compatibility? If possible, I would really like this to be gone.
- Priority calculation: The whole priority field is funky. The priority does not really have any meaning. The order does not imply importance. Why having this at all?
- Timestamps: What’s the reason for having the “T” in the timestamp? Having looked at hundreds of different log formats, I have never seen anything like that. Why doing this?
- Hostname: I am not comfortable with the whole hostname spec. I like that there is an ordering and people are supposed to use FQDNs, but there are many questions about this. To start with, in a lot of UNIX configurations, /etc/hosts contains an entry like
127.0.0.1  localhost.localdomain localhost
The second column is the FQDN (technically). Is that one that can be used? Can you make it clear that this is not what should be used? Same for 127.0.0.1 or the loopback address in general. How does a machine know whether an IP address is static or dynamic? How does a logging application know? I don’t think you will ever know that. Did you mean a private versus a public address? That might be interesting. Furthermore, it should specify which interface’s IP address to use. The interface that the message is sent out on?
- Under the section of PROCID: The text is imprecise. This number is not the process ID of the syslog process, it’s the ID of the writing process. The third paragraph talks about detecting restarted applications and somehow mixes in the syslog process. (“might be assigned the same process ID as the previous syslog process“.) This is not clear at all and very very confusing.
- MSGID: Make clear that this ID is local to the application. It’s not a global ID at all.
The biggest issue I have around the SD-ID field:
- I like that the user can extend the set of registered IDs.
- Why is this structure so complicated? Why not going with a simple set of key-value pairs? This whole structure thing is so complicated. Parsing it, you need to keep state! You need to remember the SD-ID for each SD-PARAM. Why introducing this? Just stick with simple key-value pairs. That makes parsing easier. Much easier. And it makes the events easier to produce as well.
- By keeping an explicit message field (the unstructured part), you encourage people to still log in that way. I recommend using an explicit field (or parameter) that can be used to include human readable text. Instead of this:
<165>1 2003-08-24T05:14:15.000003-07:00 192.0.2.1 myproc 8710 - - %% It's time to make the do-nuts.
use:
<165>1 2003-08-24T05:14:15.000003-07:00 192.0.2.1 myproc 8710 - message="%% It's time to make the do-nuts."
or really:
1 2003-08-24 05:14:15.000003-07:00 host=192.0.2.1 process=myproc procid=8710 message="%% It's time to make the do-nuts."
- I definitely like the consideration of some of the special fields (structured data IDs). However, they should be used as simple keys (or parameters) that have special meaning.
- Parameter – origin: What does it mean to have multiple origins IPs? Is that a syslog forwarding chain? The document does not say anything about that. Also, we already have the host field in the beginning of the syslog messages. What’s the relationship to that? Or is origin something completely different?
- Parameters – I would really like to see some use-cases for all of the IDs. Especially the sequenceId. I am assuming this is something that the syslog daemon assigns, not the logging application. Right? I think that needs to be clearer. For the sequenceId, what happens for forwarded messages? Are these IDs local? Are they forwarded along with a message? Also, how does the logging application know about the timeQuality? Or if that something that the syslog daemon assigns, how does it know?
- I would really like to see the parameters to go away and have a generic key-value extension. In addition, IANA should have a set of allowed/defined keys. The parameters should be part of those. Each key has a special meaning (semantics). There should be a whole lot of them: src_ip, user_name, etc. Each producer should be free to add additional keys, realizing that not all consumers would understand their semantics. However, the consumers could still read them.
That’s it for now… Let’s see what some of the reactions are going to be.
[tags]IANA, syslog, syslog protocol, IETF, logging[/tags]
