Splunk 9.0: Directory Naming Change

With the introduction of Splunk Enterprise 9.0, Splunk has changed the language used for certain directories in indexer-clustering. Therefore, let’s explore how this works in practice.

Some history and background

In versions of Splunk prior to 9.0, the configurations sent to the indexers would be stored in the $SPLUNK_HOME/etc/master-apps directory on the cluster manager. Configurations pushed to pre-9.0 indexers would end up in the $SPLUNK_HOME/etc/slave-apps directory. This changes with the Splunk Enterprise 9.0 release. 

There are now two possible locations for storing apps on a cluster-manager: master-apps and manager-apps. The manager-apps location is the preferred location for new installs, and the master-apps directory is maintained for backwards compatibility. Note that you can use only one of these locations at a time.

On Splunk Enterprise 9.0 indexers, apps received from a cluster manager will end up in the $SPLUNK_HOME/etc/peer-apps directory. This change is automatic and is only managed by the cluster manager, so there shouldn’t be anything you need to account for when upgrading to Splunk 9.0.

However, since the choice of location for apps on the CM is something that the user controls, I had to find out: what happens if someone were to attempt to push configuration from the wrong location? 

Pushing a bad config

One immediate question I had when reading about the configuration change: What might happen if someone were to accidentally put a new app in the incorrect directory and attempt to push a bundle to the indexers? 

The good news: Splunk figured someone would mess this up, and there is error checking in place to block a push from occurring if there are apps in both the manager-apps and master-apps directories at the same time:

To resolve this issue, the administrator just needs to make sure there is only one directory (either master-apps or manager-apps) that contains configuration files. If this does happen in your environment, you’ll hopefully catch it on the first cluster bundle push after the error is made. 

Below is a demo showing how this happens in practice. 

Migrating from master-apps to manager-apps

To perform a migration from the legacy master-apps directory to the new manager-apps directory on your cluster manager, simply move all the apps to the new manager-apps directory and push a bundle to the indexers as you normally would. Your cluster manager will now be using the new path for apps. Be sure to also update your documentation to point to the new location, and ensure any mechanisms outside of Splunk that would write to the old manager-apps directory (such as configuration management tools) point to the new location as well. 

Below is a demo showing the migration process.

Wrap up

All-in-all, this is a relatively minor change in the overall Splunk administration workflow. However, if you’re used to the pre-9.0 way of doing things, it will be a change you’ll need to get used to, but there’s no need to worry about the process accidentally breaking your Splunk environment.