When deploying Splunk Enterprise Security–both on-prem as well as in Splunk Cloud–there are several configuration optimizations you can use to improve the performance of the environment.
A notable example is the scheduler configuration. This configuration allows for more scheduled and summarization searches to run simultaneously. This is good to know since the default scheduler settings in Splunk often don’t allow the use of enough resources for these searches.
How this works with Splunk Enterprise
For on-premise Splunk Enterprise, the recommendation is to make the following changes in limits.conf to adjust the behavior of the Splunk scheduler:
This configuration means that 75% of the available search slots are used for scheduled searches. And 100% of the scheduled search slots can be used for summarization searches.
Without any customization, a search head that meets the minimum number of CPU cores for Splunk Enterprise Security (currently 16 CPU cores) would have 22 available search slots ( x + ). The default scheduler configuration would then allow for 11 concurrent scheduled searches and 5 concurrent acceleration searches.
The configuration above increases these settings to allow for 16 concurrent scheduled searches to run. All can be used for acceleration searches. This increases the number of acceleration searches that can be run simultaneously by more than three times. The increase can make the difference between these searches skipping and falling behind or not.
What to do on Splunk Cloud?
While Splunk Cloud does allow for the self-installation of apps, there are a number of configurations that you cannot include in an app in order for it to pass the app vetting process. This includes settings in limits.conf, such as the scheduler stanza noted above.
In the past, it was difficult to get this sort of change made on a Splunk Cloud Enterprise Security search head. This was because the requirement was intervention from the support team in order to make a manual configuration change. However, recent versions of Splunk Cloud introduced an option in the GUI to make adjusting this setting easy.
I first learned about this from a coworker when troubleshooting a scheduler error. So, unless I’m just living under a rock, I’m sure the existence of this feature is new information to many more of you.
Changing Scheduler Behavior
First, Navigate to Settings → Server Settings → Search preferences. The screen that appears will have two options for adjusting search concurrency. One is for scheduled searches and one for summarization searches.