Okay, this post is going to be a bit strange. It’s a quick brain dump from a technical perspective, what it would take to build the perfect security monitoring environment.

We’d need data and contextual information to start with:

Any data:

• infrastructure / network logs (flows, dns, dhcp, proxy, routing, IPS, DLP, …)
• host logs (file access, process launch, socket activity, etc.)
• HIPS, anti virus, file integrity
• application logs (Web, SAP, HR, …)
• metrics
• configuration changes (host, network equipment, physical access, applications)
• indicators of compromise (threat feeds)
• physical access logs
• cloud instrumentation data
• change tickets
• incident information

Any context:

• asset information and classification
• identity context (roles, etc.)
• information classification and location (tracking movement?)
• HR / presonell information
• vulnerability scans
• configuration information for each machine, network device, and application

With all this information, what are the different jobs / tasks / themes that need to be covered from a data consumption perspective?

• situational awareness / dashboards
• alert triage
• forensic investigations
• metric generation from raw logs / mapping to some kind of risk
• incident management / sharing information / collaboration
• hunting
• running models on the data (data science)
• • anomaly detection
  • behavioral analysis
  • scoring
• reports (PDF, HTML)
• real-time matching against high volume threat feeds
• remediating security problems
• continuous compliance
• controls verification / audit

What would the right data store look like? What would its capabilities be?

• storing any kind of data
• kind of schema less but with schema on demand
• storing event data (time-stamped data, logs)
• storing metrics
• fast random access of small amounts of data, aka search
• analytical questions
• looking for ‘patterns’ in the data – maybe something like a computer science grammar that defines patterns
• building dynamic context from the data (e.g., who was on what machine at what time)
• anonymization

Looks like there would probably be different data stores: We’d need an index, probably a graph store for complex queries, a columnar store for the analytical questions, pre-aggregated data to answer some of the common queries, and the raw logs as well. Oh boy 😉

I am sure I am missing a bunch of things here. Care to comment?