First Look: Splunk 9.0 Configuration Change Logging

By |Published On: June 15th, 2022|

One of the most requested features in Splunk has been better audit logging for changes. With the introduction of Splunk Enterprise 9.0, a new feature has been introduced for configuration change tracking. Let’s take a look at how this new feature works!

Overview

To implement the change logging feature, Splunk 9.0 introduces a new log file, configuration_change.log, and a new index, _configtracker. While this feature is enabled by default, you may need to update your indexes configuration to ensure that the _configtracker index is stored on the correct volume when you upgrade. 

The configuration_change.log file is stored in the default Splunk log directory, $SPLUNK_HOME/var/log/splunk. Indexing for this log is enabled by default in Splunk 9.0. 

Let’s explore a few changes and how they’re logged in this file and in Splunk. 

Configuration Change in SplunkWeb

Let’s start by making a simple change in SplunkWeb to observe how it is logged. I’ll start by navigating Settings -> Fields -> Field Aliases to make a new field alias. For this example, everything I’m creating is very fabricated, just so it’s easy to identify in the logs.

Review the change log

Once this change is made, it will be logged to the configuration_change.log file:

Copy to Clipboard

Since you have Splunk, rather than looking at change on the command line, let’s just search for it!

This search will show you everything in the configuration_change.log file in your environment:

Copy to Clipboard

Since we know that the change contained demo_alias, we can add this term to the search:

The search results will show the change details (expand some of the sections to see all of the relevant information):

From this example, we can see the stanza that was modified as well as the new_value and old_value of the configuration in question. 

Here’s a video demonstrating this logging in action:

What about command line changes? 

Good news! This same configuration logging is also available for changes made to the filesystem (they will be recorded when Splunk is restarted). I can see this being enormously helpful when troubleshooting Splunk issues and trying to identify when a problem occurred and why. 

Since we already have the change for our first example, let’s directly edit the configuration file (/opt/splunk/etc/users/tom/launcher/local/props.conf) on the command line:

Copy to Clipboard

Upon restarting Splunk, we’ll see that a new entry is logged to configuration_change.log:

Copy to Clipboard

Re-running our Splunk search, we’ll also see another event for this updated stanza:

Here’s a video demo showing how this works in practice for filesystem changes:

Conclusion

This configuration change logging is a feature that I’m really looking forward to seeing widely deployed across every Splunk environment. It’s been long overdue for inclusion in the product and will be a great tool for troubleshooting Splunk deployments.

About Hurricane Labs

Hurricane Labs is a dynamic Managed Services Provider that unlocks the potential of Splunk and security for diverse enterprises across the United States. With a dedicated, Splunk-focused team and an emphasis on humanity and collaboration, we provide the skills, resources, and results to help make our customers’ lives easier.

For more information, visit www.hurricanelabs.com and follow us on Twitter @hurricanelabs.