I am still playing with R to generate graphs. I have to say, after some initial frustrations, I think I start to get it. Here are some steps to generate parallel coordinate graphs in R:

Dataset < - read.table("/home/ram/foo2_200.csv", header=FALSE, sep=",") require(lattice) parallel(~Dataset,data=Dataset)

That's the very basics to generate a parallel coordinate plot. An interesting addition to this is to use a different command to generate the parallel coordinate plots:

parallel(~Dataset|Dataset$V4,data=Dataset)

This will generate n different parallel coordinates, one for each of the values in Dataset$V4 [where Dataset$V4 is the fourth column of our data].

Following is my last attempt. I wanted to change the axis labels. The y-axis I was sucessful in changing. For the x-axes, I was not able to change the labeling. I guess this is precisely the problem with R. Simple things are fairly simple to do, but if you want to change specific details, it gets quite messy quite quickly.

parallel(~Dataset,data=Dataset,varnames=c("Source","Port","Destination","Action"))

I can’t believe it. I was fighting with R (the statistical package) for a while now. All the restrictions about data types and such are driving me crazy. It constantly complains that something is not a numerical type if I try to generate a histogram, etc. Well, I just found THE solution: R Commander:

apt-get install r-cran-rcmdr
R --gui=tk
library(Rcmdr)
You are in business! I love it! It just does things for you! I am back in business and can continue writing my book!

I came accross this very well done Web Log Analysis. The author uses a 3D scatter plot to plot certain aspects of his Web server log. He uses gnuplot to do so. What I like in particular is his discussion of the output and the way he positions scatter plots to find correlated event fields.

I am finally biting the bullet. I will start to really anonymize my graphs. In order to do so, I was trying to find a tool on the Web which does that. Well, as you can probably imagine, there is non which does exactly what I wanted. So i wrote my own anonymization script. To safe you some hassle, also download the Anonymous.pm file.

This is how you use the script on a CSV file:

cat /tmp/log | ./anonymize.pl -c 1 -p user

Credits to  John Kristoff who wrote the Anonymous.pm module for Perl.

I thought I would start sharing my del.icio.us links. Especially people that are into visualization might enjoy some of the links. I will keep posting good links and discussion entries here and on secviz.org

But I thought this was kind of fun thing to do 😉

It was a bad oversitght that secviz.org did not have an RSS Feed. But now there is one! The feed contains all new content posted to the portal, including comments. Subscribe so you don’t have to check back all the time to see whether there is new content.

[tag]security visualization[/tag]

I launched a new portal that deals with visualization of log files:

http://secviz.org

The portal can only survive if people- you – take an active part in contributing content.

There are multiple resources available where community input is most welcome:

* Graph Exchange: The idea is that people can submit their graphs, explain why they think the graphs are useful, and how they generated them.
* Parser Exchange: To generate graphs, you need to parse your data. This is a place where you can submit your parsers.
* Links: A whole bunch of links around data analysis and visualization.
* Discussions: A free forum where you can start discussions around the topics of log visualization and analysis.

Let me know what you think and most importantly, submit your graphs

I am scribbling on another book chapter. This time it’s for a visualization book. I am writing about how to analyze firewall and IDS logs. I am using line graphs and treemaps to do so. Guess what tool I am using to generate all the graphs. Yes. AfterGlow.
I am not quite done with writing, but am pretty happy with the way it shapes out. The chapter is not going to be highly technical. I am not going into how to configure AfterGlow and parse log messages and such. I focus more on the process-level. It is quite an interesting experience to put something into words that you intuitively do all the time.
I am not sure when the book is actually going to come out, but I will post here when it’s available.

Yes, I am still playing around with visualization. In fact, I am writing a Chapter for a book. It’s going to be about using line graphs and treemaps to visualize firewall and IDS logs.

I just realized something that you have to be careful about when using AfterGlow’s fan-out filtering!

If you have a graph where you are going to filter based on the source-node’s fan-out, you will still see all the eventNode->targetNode pairs! It won’t restrict them to only that ones that are from the remaining source nodes! Here is an example:

My log contains thousands of line, but only these three entries should remain after doing a (-f 2) filtering:

12.108.194.82,nounce,111.222.69.44|113
12.108.194.82,block,111.222.69.44|113
12.108.194.82,pass,111.222.69.44|25

However, the graph will look as this:

What you can do is using the -p 1 option: afterglow.pl -p 1 -f 2 which will result in the desired graph:

I just came accross this review on DefCon presentations. Is published on unixreview.com. I don’t know the author, but I certainly like what she had to say about my presentation:

I’ll begin with Afterglow, by Raffael Marty, which is a visual log analysis tool. Martyýs tool runs either via Perl or Java and does some cool stuff by making text log files into understandable graphs. This type of tool can help you truly understand the big picture of what your logs are telling you without making you go cross-eyed trying to read them all. This was the second year Marty has discussed this tool, and I was very impressed by his dedication to this work.

There is actually another person out there who wrote a summary of DefCon 13 (so last year) and put some graphs online. I am impressed 😉