Optimizing Your Splunk Cloud Scheduler for Enterprise Security

By |Published On: July 20th, 2022|

Introduction

When deploying Splunk Enterprise Security, there are several configuration optimizations which can be used to improve the performance of the environment. A notable example is the scheduler configuration, which allows for more scheduled and summarization searches to run simultaneously. The default scheduler settings in Splunk often do not allow for enough resources to be used for these searches. 

How this works with Splunk Enterprise

For on-premise Splunk Enterprise, it is recommended to make the following changes in limits.conf to adjust the behavior of the Splunk scheduler:

Copy to Clipboard

This configuration means that 75% of the available search slots are used for scheduled searches, and 100% of the scheduled search slots can be used for summarization searches. 

Without any customization, a search head that meets the minimum number of CPU cores for Splunk Enterprise Security (currently 16 CPU cores) would have 22 available search slots ( <max_searches_per_cpu: 1 > x <number of CPU cores SH: 16> + <base_max_searches: 6>). The default scheduler configuration would then allow for 11 concurrent scheduled searches and 5 concurrent acceleration searches.

The configuration above increases these settings to allow for 16 concurrent scheduled searches to run, all of which can be used for acceleration searches. This increases the number of acceleration searches that can be run simultaneously by more than three times–which can make the difference between these searches skipping and falling behind or not. 

What to do on Splunk Cloud?

While Splunk Cloud does allow for the self-installation of apps, there are a number of configurations that cannot be included in an app in order for it to pass the app vetting process. This includes settings in limits.conf, such as the scheduler stanza noted above. 

In the past, it was difficult to get this sort of change made on a Splunk Cloud Enterprise Security search head, since it required intervention from the support team in order to make a manual configuration change. However, recent versions of Splunk Cloud introduced an option in the GUI to make adjusting this setting easy. 

I first learned about this from a coworker when troubleshooting a scheduler error, so I’m sure the existence of this feature is new information to many more of you (or I’m just living under a rock). 

Changing Scheduler Behavior

First, Navigate to Settings → Server Settings → Search preferences. The screen that appears will have two options for adjusting search concurrency, one for scheduled searches and one for summarization searches.

To configure the recommended settings for Splunk Enterprise Security, change the scheduled limit to 75% and your summarization limit to 100%:

Upon clicking save, you’re done. The new scheduler settings will be used going forward. Note that these changes are managed independently on each Splunk search head, so you can have different settings on your ES and ad-hoc instances.

Below is a quick demo showing you how to do this change on a Splunk Cloud search head:

Wrap Up 

While this isn’t a setting you’ll likely be changing often, knowing that the option to adjust your scheduler’s limits in the Splunk Cloud UI is quite useful, especially if you’re running into issues with skipped searches on your Enterprise Security search head. 

Happy Splunking!

About Hurricane Labs

Hurricane Labs is a dynamic Managed Services Provider that unlocks the potential of Splunk and security for diverse enterprises across the United States. With a dedicated, Splunk-focused team and an emphasis on humanity and collaboration, we provide the skills, resources, and results to help make our customers’ lives easier.

For more information, visit www.hurricanelabs.com and follow us on Twitter @hurricanelabs.