Optimizing the Splunk Cloud Scheduler for Enterprise Security
When deploying Splunk Enterprise Security–both on-prem as well as in Splunk Cloud–there are several configuration optimizations you can use to improve the performance of the environment.
A notable example is the scheduler configuration. This configuration allows for more scheduled and summarization searches to run simultaneously. This is good to know since the default scheduler settings in Splunk often don’t allow the use of enough resources for these searches.
How this works with Splunk Enterprise
For on-premise Splunk Enterprise, the recommendation is to make the following changes in limits.conf to adjust the behavior of the Splunk scheduler:
This configuration means that 75% of the available search slots are used for scheduled searches. And 100% of the scheduled search slots can be used for summarization searches.
Without any customization, a search head that meets the minimum number of CPU cores for Splunk Enterprise Security (currently 16 CPU cores) would have 22 available search slots ( x + ). The default scheduler configuration would then allow for 11 concurrent scheduled searches and 5 concurrent acceleration searches.
The configuration above increases these settings to allow for 16 concurrent scheduled searches to run. All can be used for acceleration searches. This increases the number of acceleration searches that can be run simultaneously by more than three times. The increase can make the difference between these searches skipping and falling behind or not.
What to do on Splunk Cloud?
While Splunk Cloud does allow for the self-installation of apps, there are a number of configurations that you cannot include in an app in order for it to pass the app vetting process. This includes settings in limits.conf, such as the scheduler stanza noted above.
In the past, it was difficult to get this sort of change made on a Splunk Cloud Enterprise Security search head. This was because the requirement was intervention from the support team in order to make a manual configuration change. However, recent versions of Splunk Cloud introduced an option in the GUI to make adjusting this setting easy.
I first learned about this from a coworker when troubleshooting a scheduler error. So, unless I’m just living under a rock, I’m sure the existence of this feature is new information to many more of you.
Changing Scheduler Behavior
First, Navigate to Settings → Server Settings → Search preferences. The screen that appears will have two options for adjusting search concurrency. One is for scheduled searches and one for summarization searches.
To configure the recommended settings for Splunk Enterprise Security, change the scheduled limit to 75% and your summarization limit to 100%:
Upon clicking save, you’re done. You can use the new scheduler settings going forward. Note that these changes are managed independently on each Splunk search head. This means you can have different settings on your ES and ad-hoc instances.
Splunk Search Head Demo
Below is a quick demo showing you how to do this change on a Splunk Cloud search head:
Ultimately, it’s useful to know the option to adjust your scheduler’s limits in Splunk Cloud
While this isn’t a setting you’ll likely be changing often, knowing that the option to adjust your scheduler’s limits in the Splunk Cloud UI is quite useful. This is especially true if you’re running into issues with skipped searches on your Enterprise Security search head.
If you need any assistance with your Splunk environment, don’t hesitate to get in touch! We are always happy and ready for a discussion about how we can help.
Happy Splunking!
About Hurricane Labs
Hurricane Labs is a dynamic Managed Services Provider that unlocks the potential of Splunk and security for diverse enterprises across the United States. With a dedicated, Splunk-focused team and an emphasis on humanity and collaboration, we provide the skills, resources, and results to help make our customers’ lives easier.
For more information, visit www.hurricanelabs.com and follow us on Twitter @hurricanelabs.
