March 16, 2012

Big Data Security Intelligence – nothing to see here – move along

Category: Log Analysis,Security Intelligence,Visualization — Raffael Marty @ 7:22 am

Big data doesn’t help us to create security intelligence! Big data is like your relational database. It’s a technology that helps us manage data. We still need the analytical intelligence on top of the storage and processing tier to make sense of everything. Visual analytics anyone?

A couple of weeks ago I hung out around the RSA conference and walked the show floor. Hundreds of companies exhibited their products. The big topics this year? Big data and security intelligence. Seems like this was MY conference. Well, not so fast. Marketing does unfortunately not equal actual solutions. Here is an example out of the press. Unfortunately, these kinds of things shine the light on very specific things; in this case, the use of hadoop for security intelligence. What does that even mean? How does it work? People seem to not really care, but only hear the big words.

Here is a quick side-note or anecdote. After the big data panel, a friend of mine comes up to me and tells me that the audience asked the panel a question about how analytics played into the big data environment. The panel huddled, discussed, and said: “Ask Raffy about that“.

Back to the problem. I have been reading a bunch lately about SIEM being replaced or superseded by big data infrastructure. That’s completely and utterly stupid. These are not competing technologies. They are complementary. If anything, SIEM will be replaced by some other analytical capabilities that are leveraging big data infrastructures. Big data is like RDBMS. New analytical capabilities are like the SIEMs (correlation rules, parsed data, etc.) For example, using big data, who is going to write your parsers for you. SIEMs have spent a lot of time and resources on things like parsers, big data solutions will need to do the same! Yes, there are a couple of things that you can do with big data approaches and unparsed data. However, most discussions out there do not discuss those uses.

In the context of big data, people also talk about leveraging multiple data sources and new data sources. What’s the big deal? We have been talking about that for 6 years (or longer). Yes, we want video feeds, but how do you correlate a video with a firewall log? Well, you process the video and generate events from it. We have been doing that all along. Nothing new there.

What HAS changed is that we now have the means to store and process the data; any data. However, nobody really knows how to process it.

Let’s start focusing on analytics!


  1. […] as “I think we’re in a precarious spot”. Raffael follow up recently with his post ( that speaks to me loud. Moar […]

    Pingback by Big Data Information Security Maturity Scale – Where are you on this scale? « facebookjustice — March 17, 2012 @ 11:35 am

  2. […] of Big Data Infosec at Blackhat EU 2012 (Link:PDF). Raffael followed up recently with his post (Link:Website). Moar visual analytics! Ed in this post suggested that Infosec stop using “stoplight […]

    Pingback by Big Data Infosec – Bigsnarf Open Source Solution « BigSnarf blog — March 23, 2012 @ 12:46 pm

  3. Security Management is a 2 layered cake:
    1st Layer : IDP/IPS/UTMs… real time/attack protection
    2nd Layer : SIEM, Post real time – event/threat analysis, data forensics and law compliance.
    Now the legacy RDBMS/SIEMs can’t cope in this Big Data world, so next generation “SIEMs” like Secnology will be needed.
    A 3rd layer Big Data “Hadoop” “Magic” analysis tool sounds sexy but when & at what price. Considering the SIEM still hasn’t made the middle market yet, it’ll still take a Data/Security expert to harness the analytics & make the calls.

    Comment by VMercer — November 22, 2012 @ 9:13 am

  4. > SIEM being replaced or superseded by big data infrastructure. That’s completely and utterly stupid.

    I didn’t quite understand this claim. From what I’m assuming, SIEM still has lots to offer that this new product can not do. I’ve seen customers do this and it is foolish. Regardless, I’d like to hear what you have seen.

    Comment by AlbeeTu — January 9, 2013 @ 12:54 pm

  5. AbleeTu … What I was reacting to was that everyone is looking at big data, but what big data is today is an infrastructure layer. Only a handful of companies are working ont he analytical layer of big data. That’s what we really need.
    I didn’t say SIEM was already gone. I strongly believe though that there will be a new wave of SIEMs or whatever they will be called. I am obviously very bullish on visual analytics (see pixlcloud). I have seen too many customers that are struggling with dealing with all their data in the SIEM. They have no idea what they are collecting, what the data tells them, and what to do with it. In order to process that amount of data, we need new data stores. RDBMs is not cutting it anymore (I am happy to have a long discussion with you on this). Think OLAP 2.0. We need better ways for analysts to interact with the data. Not sure I am expressing myself that well…

    Comment by Raffael Marty — January 11, 2013 @ 9:01 am

RSS feed for comments on this post. | TrackBack URI

Leave a comment

XHTML ( You can use these tags): <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong> .