April 26, 2006

Log Messages

Category: Log Analysis — Raffael Marty @ 3:53 pm

Quite a while ago, the goal of loganalysis.org was to collect log files of all kinds of devices to build up a repository for the community. Unfortunately that effort has not been too successful. I just stumbled accross a new effort driven by splunk:
www.splunk.com/base. There are quite a few syslog messages on there already. What i don’t like is that most of the messages are some kind of exceptions of some java applications. I don’t really care about those things. Well. Hopefully there are going to be more people adding logs…

April 13, 2006

Filtering vs. Prioritization

Category: Security Information Management — Raffael Marty @ 1:09 pm

I was just emailing someone who suggested a thesis on the topic of filtering event streams to get rid of false positives. This is what I replied:

Filtering seems to be the obvious approach to take in order to get to the important events in an event stream.However, filtering is not really what you want to do. You can filter all day and you still end up with a lot of stuff that you have not filtered (e.g., new things will show up and you will have to filter again). Do the math: 1Mio events a day. Assum you come up with a lot of filters that filter out 500K events. You still have 500K events left. What you need to do is prioritization. You need to have those things that are important trickle up! You can still apply filtering after that, but prioritize first!

Here is a very important concept in SIM: Don’t spend processing time on unimportant things!

April 4, 2006

Perl Performance Optimization

Category: Programming — Raffael Marty @ 4:13 pm

I was working on AfterGlow the other night and I realized that adding feature after feature starts to slow down the thing quite a bit (you need to be a genious to figure that one out!). So that prompted me to look for Perl performance analyzers and indeed I found something that’s pretty useful.

Run your perl script with: perl -d:DProf and then run dprofpp. This will show you how much time was spent in each of the subroutines. It helped me pinpoint that most of the time was spent in the getColor() call. The logical solution was to introduce a cache for the colors and guess what – AfterGlow 1.5.1 will be faster 😉

This is a sample output of dprofpp:

Total Elapsed Time = 11.69959 Seconds
User+System Time = 8.969595 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c Name
81.5 7.310 9.900 120000 0.0001 0.0001 main::getColor
29.1 2.615 2.615 116993 0.0000 0.0000 main::subnet
0.89 0.080 0.080 20000 0.0000 0.0000 main::getEventName
0.22 0.020 0.020 20000 0.0000 0.0000 main::getSourceName
0.22 0.020 0.020 20000 0.0000 0.0000 main::getTargetName
0.11 0.010 0.010 1 0.0100 0.0100 main::BEGIN
0.00 - -0.000 1 - - Exporter::import
0.00 - -0.000 1 - - Getopt::Std::getopts
0.00 - -0.000 1 - - main::propertyfile
0.00 - -0.000 1 - - main::init
- - -0.025 116993 - - main::field

March 20, 2006

Anton And Raffy @ RSA 2006

Category: Uncategorized — Raffael Marty @ 9:10 pm

I wanted to post this picture for a while. Here we go. Anton (NetForensics) and myself (ArcSight) at some of our competitors booths … Thanks for taking the pictures!

Anton And Raffy AT RSA

AfterGlow 1.5

Category: Visualization — Raffael Marty @ 6:52 pm

Another release of AfterGlow. It’s mainly a feature release where I introduce fan-out filtering. I also fixed some minor bugs around property file parsing and some error messages. Nothing major.

The fan-out filtering is really cool. So far you could only filter based on node counts. Saying for example you wanted to eliminate all the nodes which had a count of 5 or less. Now you can say you want to filter out all the nodes which have a fan out (outgoing neighbors) of 4 or less. This way you can for example identify scans, as here:

Fan Out Filtering

This was generated with:

afterglow.pl -c color.properties -p 1 -d -o 7 -g 8

Where -g is the filter for the fan-outs. The graph clearly shows that some external machines were scanning all the internal machines on one and the same port.

I am still waiting for people to send me their graphs!!

March 10, 2006

AfterGlow 1.4

Category: Visualization — Raffael Marty @ 1:12 pm

I was playing with AfterGlow and visualizing netflow data lately. I realized that some of the configurations in AfterGlow are a bit tedious. So I added a new property to address coloring independent of the node: color=”blue” for example. I also added a feature to turn off labels on a per-node type. For example label.source=0 turns off source node labels. And finally I started playing with clustering of nodes. A lot of times you end up with a huge amount of nodes and you don’t really need to know all the details of that cluster, you just want to know that there is something going on around there. So for example for IP addresses, oftentimes you want to cluster by subnets (e.g., A, B, C classes):

cluster.source=regex_replace(“(\\d\+\\.\\d+)”).”/16″ if (!match(“^(212\.254\.110|195\.141\.69)”))

This looks complicated, but it’s not that bad. Basically there are a few functions I introduced. Read it from right to left. First you check whether there is a match on the source field (we are doing cluster.source) for either of these two IP ranges. If so, you take that value and you do a regex_replace on it. Basically getting the two first octets and adding “/16″ to it. An easier example would be:

cluster.target=”bigger than 1024” if ($fields[1]>1024)

AfterGlow 1.4, which supports clusters and all the other things is now available via sourceforge.

Clustering Example

March 8, 2006

Gartner – Messing with the Acronyms again

Category: Uncategorized — Raffael Marty @ 7:01 pm

Fresh from the press and fueling the discussion on the acronyms that I started earlier. Not just authors of articles are starting to make up new acronyms, now apparently the NBAD space is being renamed to NBA (Network Bahavioral Analysis). Why do we have to make things more complicated. People have a hard enough time already keeping track of technologies, and now you need to have a special dictionary for security acronyms?

February 28, 2006

AfterGlow 1.2 and Pinning Nodes

Category: Visualization — Raffael Marty @ 1:18 pm

While working on some firewall log analysis, I started working on AfterGlow 1.2, which is going to have the capability to turn of the node labels on a per node type basis. In addition, if you turn the labels off, the node also becomes much smaller.

But what was really interesting is that while I was browsing the man pages for graphviz, I discovered that fdp has a node attribute called pin. This got me quite excited. If pin is true, that node will remain at its initial position. Don’t ask me how that would work in reality? What is the initial position? How do you define that? Not sure, but I’ll figure it out. This would get me closer to the animated graphs!

Digging a bit deaper in the graphviz documentation, I realized that neato can also pin a node, by adding a ! to the pos argument of a node:
node [ pos = "2,2!" ];

To solve the animation, one probably has to generate a graph with the -Tplain option, then get the x and y coordinates (second and third argument to the node entries) of the graph and use them as input for the next graph.

February 21, 2006

AfterGlow 2.0

Category: Visualization — Raffael Marty @ 5:32 am

I just released AfterGlow 2.0. In addition, I released AfterGlow 1.1.6, a bug fix release to the 1.x releases.

The new release is AfterGlow 2.0, which is no more written in Perl, but in Java. It generates a new kind of graph output. Whereas AfterGlow 1.x generates link graphs, AfterGlow 2.0 generates TreeMaps.

This afternoon I will be talking at the EuSecWest 2006 Conference in London. The topic is “Visual Security Event Analysis” (what else ;)). The presentation has quite a lot of examples on how you can use both AfterGlow 1.x and AfterGlow 2.0.

I am looking forward hearing back from you with screenshots and use-cases of how you are using graphs to do security analysis.

AfterGlow 2.0 Output

February 18, 2006

Intrusion Detection Systems in 2006

Category: Uncategorized — Raffael Marty @ 12:32 pm

Can you tell that I was travelling again? Gives me a chance to catch up with the security magazines that pile up on my desk. And I keep getting disappointed. Well, there were a couple of good articles I read. One from Ed Skoudis about how to secure yourself against spyware. But most of the articles are horrible.

The first thing I found is in the Information Security Magazine. Somebody had a comment about Ed Skoudis and Mike Poor’s article on “IPS: Reloaded”. This person claims that in the old world, IDSs signatures had to be tuned, but in the new world of IPSs, that’s not necessary anymore. In his words: “IPS should not be judged with old IDS standards”. So what does this guy think IPSs do different than IDSs? Do you really think that for example the CISCO IPS is a completely new product and is not based on the old CISCO IDS code at all? What about all the other IPSs? I can guarantee you that you will have to spend as much time (if not more) to tune IPS signatures as you had to spend tuning your IDS. If IPS really had the magic sauce, why would IDSs not adopt that? Forget it!

In fact, this brings me to another thought that I had while I was walking the floor at the RSA conference in San Jose this week. There are all these new companies that I have never heard of. They are presenting solutions for all kinds of problems, ranging from insider threat detection to identity management. I spent quite some time trying to understand what they are doing. What I have seen is quite disappointing. Take an insider threat management company and check what they are doing. Well, they can detect credit card records on the wire, alert you on transmissions of social security numbers (SSN) or patient health records. Sounds great. But do you know what they are doing? Right. They basically take a NIDS sensor, apply some signatures which look for SSNs or credit card numbers. In fact, one of the companies showed me their signature definition and this is what you had to enter to detect an SSN:

\d\d\d-\d\d-\d\d\d\d

Wow! Have they ever heard of regular expressions? What about:

\d{3}-\d{2}-\d{4}

This was not their worst example! Anyways. My point is that there are all these new companies that claim amazing technology, but if you look under the hood, you realize that we had the technology for YEARS! Refurbish your NIDS and you are in great shape! Why have the NIDS vendors not jumped on the wagon? I don’t know. By the way, it’s not just the insider threat companies, but also other companies. One of them sniffs the wire and decodes all kinds of application protocols to attribute user identities to IP transactions. Again, I can solve the same problem with a Sniffer. I don’t even need a NIDS for that! [Believe me, I have tried it!]

Granted, there are some new and cools things. For example companies that let you register documents and then they detect them on the wire in any variation. For example, I register my word document. Now if someone takes the document and takes a pragraph out of it or pastes it into Excel, they are still capable of detecting that the document is on the wire. That’s pretty cool!