Dynamic thresholding using standard deviation is a common method we used to detect anomalies in Splunk correlation searches. However, one of the pitfalls with this method is the difficulty in tuning these searches. This is where the wonderful streamstats command comes to the rescue.
This Splunk tutorial will cover why tuning standard deviation searches is different from using a static threshold, how to use streamstats, and how we can use streamstats to get immediate feedback on alert volume.
Tuning Using Streamstats
1. Understanding the problem
With a static threshold search that runs over 60 minutes, calculating alert volume over 30 days is as simple as running the count by 60 minutes over 30 days. This is different with a dynamic threshold.
Typically, a standard deviation search will calculate a threshold based on the last 7 to 30 days to compare against the last hour of data. Running the same search to see approximately how many notables would be generated in 30 days will calculate the threshold differently than when it runs as a correlation search.
When running a correlation search, the threshold is based on historical data. Using the same search to calculate the alert volume for the whole 30 days the threshold will be based on historical, current, and future data for any given hour but the last.
This is where we can use streamstats to calculate the threshold based on the last 30 days for any given hour.
Still confusing? Let’s take a look at a few examples.
2. What does streamstats even do?
To understand how we can do this, we need to understand how streamstats works. In my experience, streamstats is the most confusing of the stats commands. I find it’s easier to show than explain. Let’s start with a basic example using data from the makeresults command and work our way up.
Example 1: streamstats without options