Update splunk.secret Without Breaking Your Production Environment

What is splunk.secret?

To start off, you may not know exactly what the splunk.secret is. Put simply, it is the encryption key used by Splunk for most passwords that you enter into most configuration files. When Splunk detects a plaintext password, it will encrypt the password using the splunk.secret key. You can tell that a password has been encrypted when the password string begins with “$0-9$”—this value is used by Splunk to determine if the password has been encrypted.

To accompany this blog post, I have also created a video (below) for our audiovisual learners. It is recommended that you read through the post first before viewing, as it only covers how to change it without talking about the scope and impact.

Why would you want to change splunk.secret?

The primary reason you would be changing your splunk.secret is to standardize the encryption key across all Splunk Enterprise hosts in your environment. This is highly recommended when you want to deploy any password configuration from your Deployment Server. If you do not use the same splunk.secret across hosts, you will be required to deploy the password in plaintext. Splunk may not encrypt passwords in passwords.conf which are downloaded from the Deployment Server, but it will encrypt other passwords like those in server.conf. If you manually encrypt the password, it will be re-downloaded in plaintext the next time the checksum changes on the Deployment Server. However, you should avoid deploying plaintext passwords — instead you should opt to deploy encrypted passwords, which is one of the goals of this blog post.

In a Search Head cluster, all members must have the same splunk.secret. This action should occur automatically during the initial setup of the cluster by replicating the splunk.secret from the captain to the members. If you are converting to a cluster, it would be beneficial to perform this step beforehand so you can ensure there won’t be any mismatches of stored passwords which are already encrypted.

By leveraging the same splunk.secret across all of your hosts you gain the ability to deploy encrypted passwords, ensure that all passwords are stored in an encrypted state, and to decrypt any password in your environment with one splunk.secret.

Preparing to Change the splunk.secret

Before making any changes it is important to understand the scope of the splunk.secret. Remember that it is primarily used to encrypt plain text passwords stored in your configuration files. This means that when you change the splunk.secret, all currently encrypted passwords will not work with the new encryption key. You will need to have all of the encrypted passwords in plaintext before changing the splunk.secret.

In the next section, I will provide a link to a guide that will walk you through how to decrypt a password encrypted by the splunk.secret — if you already have the passwords stored outside of Splunk you can skip that step. Encrypted passwords are used most commonly in passwords.conf (modular inputs), authentication.conf (bindDNpassword for LDAP authentication), and server.conf (pass4SymmKey and sslPassword). Encrypted passwords can also exist in: web.conf, inputs.conf, and outputs.conf. Splunk will look for passwords in these files which do not start with ‘$0-9$’, if it does not see this prefix to the password string it assumes it is unencrypted and proceeds to encrypt.

You should make special note of three configuration parameters in particular. First, bindDNpassword in authentication.conf as you may lose access to Splunk if you do not have a Splunk account and encounter an LDAP issue after changing the splunk.secret. Second, Splunk will not function properly if you do not provide the correct certificate password (‘sslPassword’ parameter in server.conf) because the server will not be able to read it’s own internal server certificate–meaning the GUI and most Splunk commands will fail. Third, if the ‘pass4SymmKey’ under the [general] stanza is wrong, the following authentication between these Splunk services will fail as it is not the same on both hosts:

• License master and its license slaves
• Members of a cluster
• Deployment server and its deployment clients

If you are using the default pass4SymmKey (from [general] stanza) and sslPassword (from [sslConfig] stanza), you can remove these lines in /etc/system/local/server.conf. Splunk will regenerate those two parameters on the next restart of Splunk using the default values from etc/system/default/server.conf–’changeme’ for pass4SymmKey and ‘password’ for sslPassword. With all this in mind it becomes clear that you will need to outline a plan before making any changes to ensure the change works with minimal downtime.

I recommend using the following approach:

1.) Create a document that lists the following components (per server):

• Splunk Server, New and Current splunk.secret, Encrypted Password, Unencrypted Password, and Password Configuration/Stanza Location

2.) Find all of the information to fill in this document

a.) Find encrypted passwords:

• Record the context (file location, stanza, parameter)

b.) List the splunk.secret:

c.) Decrypt all of the passwords and store them in the document:

• We have a script you can use to quickly decrypt Splunk passwords. If you have pip installed, you can run pip install splunksecrets. From here you can store the splunk.secret in a file and run splunksecrets --splunk-secret=file_name_here -D
• You can also decrypt passwords in newer versions of Splunk using the command splunk show-decrypted --value 'encrypted_pass_here'
• For older versions of Splunk, you can refer to this blog post: https://www.hurricanelabs.com/splunk-tutorials/make-splunk-do-it-how-to-decrypt-passwords-encrypted-by-splunk
• You can skip this step if you already have the passwords stored in plaintext outside of Splunk
• Note: ‘sslPassword’ in server.conf can not be decrypted

3.) Create a full backup of your current Splunk config

• Do not use a Splunk diag as this will not save your splunk.secret

4.) Pick the one splunk.secret you will be using and edit the current splunk.secret on each host

• You may need to add write permissions before editing the file, make sure to set it back to read-only afterwards

5.) Put all of the unencrypted passwords on their respective hosts in the correct apps/stanzas

• This is where your document will come into play
• Make sure you replace every encrypted password with the plain text version from your document
• Double check your work before proceeding, pay special attention to the app and stanza context
• You can use the following command from earlier to make sure you didn’t leave any encrypted passwords in place:

6.) Restart Splunk on each host

7.) Verify your changes worked by checking all of the following:

• If you can access the Splunk GUI
• If you can still run splunk commands that require authentication
• There is a valid connection to license master / cluster / deployment server
• If data from modular inputs is coming in
• If LDAP authentication works
• If all passwords are encrypted (Note: passwords for some modular inputs may need to be re-submitted through the app’s setup page in order to be encrypted)

The splunk.secret is located by default in $SPLUNK_HOME/etc/auth/splunk.secret. If you want to generate a new splunk.secret you can do so by removing the splunk.secret file and restarting Splunk. If you have the secret value you wish to use already, you simply edit the file with that value and restart Splunk.

Conclusion

Provided you followed the plan outlined, your splunk.secret is now standardized across all of your enterprise hosts and your passwords are encrypted. Your next steps here would be to migrate passwords to your Deployment Server where appropriate — for example your authentication and authorization configuration files are a great candidate if you use LDAP. This is now as simple as copying those stanzas onto the deployment server as deploying the encrypted password will just work.

Example Document

Splunk Resources:

• Splunk Docs: Deploy secure passwords across multiple servers
• Splunk Docs: Server.conf Spec Sheet
• Splunk Docs: How deployment updates happen
• Hurricane Labs Tutorial: How to Decrypt Passwords Encrypted by Splunk