Splunk 7.1+: Performing a Splunk Password Reset

We’ve all been there and done that – forgetting or otherwise losing access to the administrative credentials of a product we’re using. Whatever the reason, sometimes you’re in a situation where you need to reset this type of account on a system. If this application is Splunk, you’re in the right place.

These steps assume you have access to the operating system/file system of the server running Splunk either as the Splunk user or as the root user. If you do not have this level of access, you will need to be able to access the operating system before you perform password recovery on the Splunk software itself.

In versions of Splunk preceding 7.1 (versions 7.0.x, 6.6.x, and earlier), resetting the admin password was rather trivial. It would look something like this:

• Move the $SPLUNK_HOME/etc/passwd file to a backup location, such as renaming it to passwd.bak
• Restart Splunk
• Log in with admin/changeme
• Reset the admin password, and then merge the newly created etc/passwd file (containing only the admin user) with the backup file (remove the line for the admin user from the backup file first)

With the introduction of Splunk 7.1, the default admin/changeme account has been removed. This means that this method of password recovery no longer works. If you attempt to restart a 7.1 install without a passwd file, you will be greeted with the following message in SplunkWeb when trying to log in:

Of course, it’s rather difficult to make a user in the traditional methods of either applying the Splunk add user command or by adding one through the webui if you do not have working access yourself. Fortunately, even in Splunk 7.1, all hope is not lost.

There are two approaches that will work in this case: using user-seed.conf (recommended) or resetting the password hash to a known one. Let’s explore both options.

To get started, the video below will show you a quick demo of the differences between Splunk 7.1 and later (in this case, 7.3) and previous versions of Splunk, such as 7.0.

Using user-seed.conf

In Splunk 6.5, the user-seed.conf file was added to Splunk docs as a mechanism for specifying the default credentials for a new Splunk installation. This file is only relevant when $SPLUNK_HOME/etc/passwd is not present, so using it will be similar to the legacy password recovery method of moving the passwd file, restarting Splunk, and logging in with admin/changeme.

Note: user-seed.conf enforces password complexity rules. If you use a simple password (such as changeme), the password will not be set by this method (unless you use a password hash instead).

To reset the password using this method, follow the steps below.

If you prefer a more audio-visual route, I’ve created a video tutorial for you as well. Again, this will show you how to use the Splunk recommended method of resetting a password using user-seed.conf.

1. Move the existing $SPLUNK_HOME/etc passwd file to a backup location.

One location option would be $SPLUNK_HOME/etc/passwd.bak:

2. Generate a password hash to use in user-seed.conf.

You can also use a plaintext password in user-seed.conf, but putting plaintext passwords in config files should be avoided whenever possible.

There are two methods that can be used to generate a password hash. In Splunk 7.1 and later, there is a built in command to do so, which is Splunk hash-passwd.

Alternatively, you can create a sha-512 password hash using the Linux mkpasswd command:

Note: If you run the splunk hash-passwd or mkpasswd commands multiple times, you will get different output even if you use the same password. For example, both the line above and the one featured directly below are password hashes for the legacy Splunk default password, “changeme”.

3. Create a user-seed.conf file in $SPLUNK_HOME/etc/system/local containing the username and password (or password hash) you would like to use.

Note: Specifying a simple password such as “changeme” in user-seed.conf will not work due to complexity validation.

4.) Restart Splunk.

5.) A passwd file will be generated, and you will be able to log in successfully.

6.) The user-seed.conf file has been deleted by Splunk.

This is good, especially if you specified a password in plaintext.

7.) Merge new etc/passwd file with the backup file.

Your final step will be to merge the newly created etc/passwd file (containing only the admin user) with the backup file, removing the line for the admin user from the backup file first.

Resetting the password hash

As an alternative, you can reset Splunk login credentials by modifying the Splunk password file manually. This approach works in every recent version of Splunk, but is generally not recommended by Splunk support due to the risks of errors associated with these manual changes.

Splunk passwords are stored in a hashed form in $SPLUNK_HOME/etc/passwd. This file uses the same format as the /etc/passwd file you would find on any typical Linux system. However, the hashed passwords for Splunk are stored directly in the passwd file as opposed to in an equivalent to the /etc/shadow file.

A sample Splunk passwd file would look something like this:

In this example, there are two Splunk users:

• Admin – the default admin user with full permissions
• Tom – a normal Splunk user without elevated permissions

Let’s say we don’t know the password for the admin user, but don’t want to break Tom’s access. We’ll handle this by updating the password hash in the passwd file for the admin user.

The following steps will help you complete this process.

Again, feel free to follow along with the video I’ve created for this section as well.

1. Generate a new password hash using the mkpasswd command.

Note: If this package doesn’t exist, you may need to install this. On my Ubuntu machine, this was accomplished by running apt install whois.

Fortunately, the error message was self-explanatory:

2. Using mkpasswd, generate a sha-512 hash.

3. Replace the existing hash.

You will do this in the $SPLUNK_HOME/etc/passwd file with the new hash you just generated.

4.) Restart Splunk.

5.) Once Splunk restarts, you will be able to log in successfully.

In Conclusion

The password hashes used are portable, so you don’t even need to generate the password hash; in a pinch, the hashes on this page should work for resetting your password to changeme. Just don’t leave it that way, since it’s insecure.

Hopefully these steps will be able to help you out if you ever lock yourself out of your Splunk instance and forget the keys, or if you need to take over management of a Splunk instance where no one has administrative credentials.

Happy Splunking!