Splunk 9.0: Configuration Change Tracking
With the introduction of Splunk Enterprise 9.0 comes the addition of a new feature for the tracking of configuration change logging. This is great because one of the top feature requests in Splunk is for better audit logging for changes.
With that in mind, let’s take a look at how this new configuration change tracking feature works!
Overview
To implement the configuration change logging feature, Splunk 9.0 introduces a new log file, configuration_change.log, and a new index, _configtracker. While this feature is enabled by default, you may need to update your indexes configuration to ensure that storing _configtracker occurs on the correct volume when you upgrade.
The configuration_change.log file is stored in the default Splunk log directory, $SPLUNK_HOME/var/log/splunk. Indexing for this log is enabled by default in Splunk 9.0.
Let’s explore a few changes and how they’re logged in this file and in Splunk.
Configuration Change in SplunkWeb
Let’s start by making a simple change in SplunkWeb to observe the logging. I’ll start by navigating Settings -> Fields -> Field Aliases to make a new field alias. For this example, everything I’m creating is a fabrication, just so it’s easy to identify in the logs.
Review the change log
Once this change occurs, it will be logged to the configuration_change.log file:
Since you have Splunk, rather than looking at change on the command line, let’s just search for it!
This search will show you everything in the configuration_change.log file in your environment:
Since we know that the change contained demo_alias, we can add this term to the search:
The search results will show the change details (expand some of the sections to see all of the relevant information):
From this example, we can see the stanza that was modified as well as the new_value and old_value of the configuration in question.
Here’s a video demonstrating this logging in action:
What about command line changes?
Good news! This same configuration logging is also available for changes made to the filesystem–the recording of these changes will occur when you restart Splunk. I can see this being enormously helpful when troubleshooting Splunk issues and trying to identify the timeframe of a problem occurring and the why.
Since we already have the change for our first example, let’s directly edit the configuration file (/opt/splunk/etc/users/tom/launcher/local/props.conf) on the command line:
Upon restarting Splunk, we’ll see the logging of a new entry to configuration_change.log:
Re-running our Splunk search, we’ll also see another event for this updated stanza:
Here’s a video demo showing how this works in practice for filesystem changes:
Conclusion
This configuration change logging is a feature that I’m looking forward to seeing widely deployed across every Splunk environment. It’s been long overdue and will be a great tool for troubleshooting Splunk deployments.
About Hurricane Labs
Hurricane Labs is a dynamic Managed Services Provider that unlocks the potential of Splunk and security for diverse enterprises across the United States. With a dedicated, Splunk-focused team and an emphasis on humanity and collaboration, we provide the skills, resources, and results to help make our customers’ lives easier.
For more information, visit www.hurricanelabs.com and follow us on Twitter @hurricanelabs.
