Over the course of the national competition on November 22-24, we collected more than 9TB of raw data, plus about 4TB of data for the regional events! One of the tools we used to help us deal with all that data was Splunk software.
I’ve had vast experience working with Splunk, from Managed Splunk Services customers at Hurricane Labs to building my own Splunk administration training, all of which has helped me use Splunk to enhance this event.
Just in case you missed them, Parts 1, 2, and 3 highlight other cool aspects of the event too!
All the Data
In addition to the Google Drive audit data discussed in Part 3 of this series, we collected an enormous amount of data from the competition systems throughout the competitions. We’re currently working to make this data available for public release and research; we believe it may be one of the most comprehensive and complete data sets ever made available.
Once this is published, this dataset will be available for download on http://mirrors.rit.edu/cptc. I will be making this available as Splunk frozen buckets, which can be imported into your own Splunk instance and searched.
I’ll also be writing up instructions on how to use this data, as well as some write-ups from other Hurricane Labs employees on interesting findings they’ve located in the data over the coming months. This will be a great source of training data for identifying attacker activity by searching in Splunk.
Some of the data that will be made available include:
- Linux Bash History
- Windows Powershell command execution history
- Windows Security, System, and Application logs
- Suricata IDS logs
- DNS, HTTP, TCP, and UDP metadata via the Splunk App for Stream
- Regular snapshots of system state via system reporting tools (eg, ps, lsof, netstat, etc. on Linux hosts)
We’ve seen some exciting research come from the 2018 dataset, such as this one on characterizing attacker behavior. I’m looking forward to seeing what else can be discovered with the new data we’ve collected from 2019.
Resources for Future Competitors