Determining available resources on the Splunk Search Head
Do you know how many CPUs and resources are available on your Splunk search head? If not, this tutorial will show you the way.
Let’s get started!
Validating resource availability on your on-prem Splunk Enterprise Splunk search head
When troubleshooting Splunk search performance, a common issue is validating that the correct resources are available. Fortunately, for on-premise Splunk Enterprise, this can be done easily through the Monitoring Console.
To do this, first navigate to: Settings -> Monitoring Console.
Here, you will see the amount of memory and CPU cores in the upper left corner:
Figuring out the size of your Splunk search head on Splunk Cloud
This approach will work on any Splunk Enterprise host. Also, this is the case regardless of whether or not the Monitoring Console configuration is in a distributed manner on another host. Note that you just need to have administrator permissions to see this option.
However, on Splunk Cloud Platform, this information is not available. The Monitoring Console is unavailable–and its replacement is the Cloud Monitoring Console app–which doesn’t convey this information. Fortunately, there are some internal logs within Splunk that can help us figure out the sizing of our search head. We can then use that information to deduce the AWS instance size that Splunk Cloud is using for your stack.
Begin by running this search:
Note: this search relies on startup messages that occur when Splunk restarts. Your instance must have been restarted in the past 30 days in order for these logs to appear.
In your search results, you will see loader events from splunkd.log. These will contain the following information:
In these events, we’ll see a few different search heads, with varying specifications. One host has 36 virtual CPUs, and the other has 72 virtual CPUs:
From here, we can reference the AWS instance types page to deduce what type of system matches those specifications. At the time of writing (May 2022), it appears that Splunk Cloud generally uses C5 instances for search heads in Splunk Cloud.
Looking at the C5 instance types page, we can see that there are two instances that match the CPU/memory combinations:
- c5.9xl = 36 vCPU and 72GB of RAM
- c5.18xl = 72 vCPU and 144GB of RAM
Since these match the startup messages in Splunk, it’s a pretty good assumption that our search heads in this deployment are c5.9xl and c5.18xl instances, respectively.
Curious about other instances in your environment (such as indexers)? You can do the same type of search by simply changing the host entry from host="sh-*.splunkcloud.com" to something like host="idx-*.splunkcloud.com" instead. Note that these are typically i3 or i3en instances in most current Splunk Cloud stacks. Also, if you’re trying to identify the resources on any system sending data to Splunk, this approach works for universal forwarders too.
Happy Splunking!
Having a quick way to determine what CPU and memory resources are available on your Splunk search head can help you be better informed when troubleshooting potential issues in your environment.
About Hurricane Labs
Hurricane Labs is a dynamic Managed Services Provider that unlocks the potential of Splunk and security for diverse enterprises across the United States. With a dedicated, Splunk-focused team and an emphasis on humanity and collaboration, we provide the skills, resources, and results to help make our customers’ lives easier.
For more information, visit www.hurricanelabs.com and follow us on Twitter @hurricanelabs.
