Deploying the Splunk Universal Forwarder on Linux

Clicking the download link will take you to a page that automatically downloads the installer of choice. One useful tool (conveniently placed in the “useful tools” section) is the “download via command line” option, which gives you a wget link that can be pasted into a terminal to download the installer directly on a Linux host.

When downloading a Universal Forwarder, pay attention to the architecture that is supported by the package (indicated in gray). Additionally, be aware of the kernel versions that are supported as well. Fortunately, this is generally much less of an issue for Linux hosts than Windows ones, with the possible exception of 32-bit Linux UFs not being supported by any currently supported version of Splunk. 

If you’re a Hurricane Labs Managed Splunk Services customer, our support team can advise you on what packages are best suited for your environment and provide the package(s) if you don’t have a Splunk account available. 

Gather Required Information

When installing this, there are multiple options, which we will explore individually below. In order to proceed with any option, you’ll want to first have the following information:

• Deployment Server: This is the host in your Splunk environment that manages configuration on all of your universal forwarders. This should be a DNS CNAME whenever possible to make future updates or server migrations easier. We do not recommend specifying the IP address of a deployment server when applying this configuration. 
• Username and password: This should be a unique username and password that will be configured on the Universal Forwarder and used in the event of any configuration changes or troubleshooting needed in the future. In versions of Splunk preceding 7.1, this was automatically set to admin/changeme, but this is now a required parameter due to security concerns around a default password. 

Important Security Note

It is generally best practice to avoid running the Splunk Universal Forwarder as root and instead install it as a user that is only granted permissions to access the files that are needed to be recorded by Splunk. This can present a monitoring challenge, however, as many of the log files you may want to collect with the Universal Forwarder are not accessible by default by non-root users. There are mechanisms to work around this, such as file system ACLs, but they are beyond the scope of this tutorial.

For the sake of simplicity, these examples will show the Universal Forwarder being installed as root. 

Using the Package Installation

These next two examples will show how to install the Universal Forwarder using one of the packages that are available from Splunk. These options are mutually exclusive, as which one you choose will depend on what package format is appropriate for your Linux distribution (eg, .rpm packages are for RedHat-based distributions, and .deb packages are for Debian-based distributions). 

The next two sections will cover each of these two installation methods. Pick the one that’s appropriate for the platform where you’re deploying the Universal Forwarder. 

RPM Installation 

The RPM installation will be used on RedHat-based Linux distributions, such as RedHat Enterprise Linux (RHEL) and CentOS.

The following video will demonstrate this installation method.

Start by downloading the .rpm installer from Splunk. For this example, I used the wget link. Then run the following rpm command to install the UF (the filename will change based on the version of the UF that you downloaded): rpm -ivh splunkforwarder-8.2.3-cd0848707637-linux-2.6-x86_64.rpm

By default, the RPM installer will install the UF to /opt/splunkforwarder

The whole process looks like this on my test instance:

Next, start the Splunk Forwarder: /opt/splunkforwarder/bin/splunk start --accept-license

You’ll be prompted to specify an administrator username and password; this is the account that was specified above for troubleshooting the UF. It does not (and should not) need to be an account that already exists on the system.

Finally, enable the Universal Forwarder to start on boot: /opt/splunkforwarder/bin/splunk enable boot-start -systemd-managed 0

Note: if you’re not running the Universal Forwarder as root, you can specify a -user argument to this boot-start command to ensure the UF process starts as the correct user. The boot-start command will need to be run as root in order to create the necessary files. 

At this point, the Universal Forwarder installation is complete, and you can move on to the next section: Configuring the Deployment Server. 

DEB Installation 

The DEB installation will be used on Debian-based Linux distributions, such as Debian and Ubuntu.

The following video will demonstrate this installation method.

Start by downloading the .deb installer from Splunk. For this example, I used the wget link. Then, run the following rpm command to install the UF (the filename will change based on the version of the UF that you downloaded): dpkg -i splunkforwarder-8.2.3-cd0848707637-linux-2.6-amd64.deb

By default, the dpkg installer will install the UF to /opt/splunkforwarder 

The whole process looks like this on my test instance:

Next, start the Splunk Forwarder: /opt/splunkforwarder/bin/splunk start --accept-license

You’ll be prompted to specify an administrator username and password; this is the account that was specified above for troubleshooting the UF. It does not (and should not) need to be an account that already exists on the system.

Finally, enable the Universal Forwarder to start on boot: /opt/splunkforwarder/bin/splunk enable boot-start -systemd-managed 0

Note: if you’re not running the Universal Forwarder as root, you can specify a -user argument to this boot-start command to ensure the UF process starts as the correct user. The boot-start command will need to be run as root in order to create the necessary files. 

At this point, the Universal Forwarder installation is complete, and you can move on to the next section: Configuring the Deployment Server. 

Using the Tarball (tgz) Installation Method

For most clients, you’ll already have some mechanism that manages packages on your Linux hosts, and using the RPM or DEB methods will make the most sense. However, Splunk also produces a tarball (tgz) of the Splunk Universal Forwarder package that can also be used. This is my preferred method for installing the UF because it’s quick and easy to use, and it lends itself well to being adapted to different environments, deployment methods, and scripting.

The following video will demonstrate this installation method.

Start by downloading the .tgz installer from Splunk. For this example, I used the wget link. Then, extract the .tgz file to the location where you want to run the Universal Forwarder. Generally, I’ll pick the default location of /opt/splunkforwarder. The following commands can be used to accomplish this (assuming that the UF package is downloaded to /tmp): tar -zxf /tmp/splunkforwarder-8.2.3-cd0848707637-Linux-x86_64.tgz -C /opt 

Note: most versions of tar support the -C argument to specify a directory to extract the tarball. If your version of tar doesn’t support this argument, you can also switch into the directory where you’re looking to extract the UF package. 

The whole process looks like this on my test instance:

Next, start the Splunk Forwarder: /opt/splunkforwarder/bin/splunk start --accept-license

You’ll be prompted to specify an administrator username and password; this is the account that was specified above for troubleshooting the UF. It does not (and should not) need to be an account that already exists on the system.

Finally, enable the Universal Forwarder to start on boot: /opt/splunkforwarder/bin/splunk enable boot-start -systemd-managed 0

Note: if you’re not running the Universal Forwarder as root, you can specify a -user argument to this boot-start command to ensure the UF process starts as the correct user. The boot-start command will need to be run as root in order to create the necessary files. 

At this point, the Universal Forwarder installation is complete, and you can move on to the next section: Configuring the Deployment Server. 

Configuring the Deployment Server

Now that the Universal Forwarder is installed, you’ll want to configure the Universal Forwarder to connect to the deployment server. This allows the Universal Forwarder to retrieve configuration details, such as what log files should be monitored and where the data should be sent to be indexed. 

There are three approaches to setting the deployment server that we’ll look at:

• Using the deploy-poll command
• Creating a deploymentclient.conf file
• Configuring a deployment app 

To configure the deployment server, you’ll need to know the name of the deployment server and the port for the Splunkd process on the deployment server (which is almost always TCP/8089). I recommend that you configure the deployment server to use a DNS CNAME, such as splunkdeploy.your-fqdn.com, as opposed to specifying the server name directly, as this makes it way easier to update this in the future. For similar reasons, I don’t recommend specifying an IP address for the deployment server, as this becomes potentially even more difficult to update later if needed. 

Using the deploy-poll command

The deploy-poll command can be specified as an argument to the splunk executable for creating a deployment app. This effectively creates a deploymentclient.conf file in $SPLUNK_HOME/etc/system/local, which we’ll cover in the next section. 

The deploy-poll command requires authentication using the username and password that you created when the Universal Forwarder was installed. If you already forgot that password, you can use one of the other two options below or check out my tutorial on Performing a Splunk Password Reset. 

The syntax for the deploy-poll command is as follows: $SPLUNK_HOME/bin/splunk set deploy-poll <deployment.server.fqdn>:<port>

For example, if you have the Splunk Universal Forwarder installed to /opt/splunkforwarder, and your deployment server is named splunkdeploy.customerscallnow.com using port 8089, you’d run the following command: /opt/splunkforwarder/bin/splunk set deploy-poll splunkdeploy.customerscallnow.com:8089

You’ll be prompted to enter the username and password for the Universal Forwarder to complete the process.

As noted above, this process creates a deploymentclient.conf file in $SPLUNK_HOME/etc/system/local:

Once this is done, restart the Universal Forwarder, and it should begin connecting to the deployment server shortly after the restart is complete.

$SPLUNK_HOME/bin/splunk restart

The video below provides a demonstration of how this looks in practice.

Creating a deploymentclient.conf file

The functional equivalent of using the deploy-poll command above is to simply create the appropriate configuration file (deploymentclient.conf) in $SPLUNK_HOME/etc/system/local. 

The syntax of this file is as follows:

[target-broker:deploymentServer] targetUri = <deployment.server.fqdn>:<port>

For example, if you have the Splunk Universal Forwarder installed to /opt/splunkforwarder and your deployment server is named splunkdeploy.customerscallnow.com using port 8089, you’d create a file at /opt/splunkforwarder/etc/system/local/deploymentclient.conf with the following content: 

[target-broker:deploymentServer] targetUri = splunkdeploy.customerscallnow.com:8089

Once this is done, restart the Universal Forwarder, and it should begin connecting to the deployment server shortly after the restart is complete.

$SPLUNK_HOME/bin/splunk restart

The video below provides a demonstration of how this looks in practice.

Creating a deployment app

This approach for configuring the deployment server is the best–but also the most complicated–approach. It allows for the deploymentclient.conf configuration to be completely managed by the deployment server, which can be useful if you need to change the name of the deployment server later (but you used a CNAME like I suggested, so that’s not an issue, right?). 

If you’re going with this approach, it’s important to name the deployment app on the Universal Forwarder to match the name of the deployment app on the deployment server. When the Universal Forwarder connects to the deployment server for the first time, it’ll replace the app that you created with the one from the deployment server (assuming the hash of the apps are different), and that app will then be managed via the deployment server moving forward. 

If you’re familiar with creating Splunk apps for managing configuration, this should be a simple process where you create an app that contains a deploymentclient.conf file, with the directory structure as shown below:

If you don’t have a sample app available, you can use the following base64 tarball as a starting point:

H4sIAAAAAAAAA+2WTXPTMBCGc9av0JmhjvwNmdITBw7cOpwynszGVlJNZMlIckv/PWuXIf0I7rSE MAz7HCxLK1u2X++rBa1Xjey0vW2lCbVWeJzPjotASiHGVjxtx/M4i7MkKURZljMRp2WRznh+5Oc4 SO8DOM5nztowNe+5+D8KHNJf2xr08f6CF+if56P+uSgE6X8KJvR/PBzV1mxes8YgcJFlv9Q/ztK9 /gnOS+JUFDMujv2yh/jP9V/uVT67k7libImfZCvD2drZnXSL/ZRL6a6lq9hd/ItT/AM/34cjP8aj zdfGXCzOO+vCBWN/+xWJCSbyH7ru1Sn/gDH/J/w/Rc9/6P9FngjK/1Ow7KDewVZWrL6S9W61sW7V dw0E6TG3N6C9RD9QBr+S1hXDJkgMSANrLRsM9apiyq+ulVc48vMaHGrB4I3h/ihbaugNroQeAn24 sg5Dn3rnVA1G8s+w9qyRvnaqC8oaDH6UG+h14Ac3Iz4c1LZ3MM7GZx9GjKyDMtuhewOukc7zYPll p3uzixj6k7+7dRwJ8qaD+d/KAPgLwJFKwJfX/3meUf1/Eqb1HzeCaOj+zhrP+X8i4kf6l7kg/z8J SzTiupZ+cHsnoeELvuRvePWW3ziFVj90oWmV4RWT34aKDif6Wx9ki26OJcL8x94wH7eG6p6/vouS KGWtbYJqhy0gLtIifZ9mZRzlcVoUaAmCqkOCIAiCIAiCIAiCIAiCIAiC+AN8B0v5AN0AKAAA

To extract this file, run the following command:

cat | base64 -d | tar -zx -C /opt/splunkforwarder/etc/apps

The terminal will pause, waiting for more input. Paste the base64 block from above, and then press CTRL+D.

This will create an all_deploymentclient app with a template deploymentclient.conf file that you can update to match your configuration. Once you’ve created this app, use the following command to create a new base64 tarball you can use on other systems. 

#run from the directory where the app exists, typically
/opt/splunk/etc/apps
tar -zc | base64

Once the app is in place, restart the Universal Forwarder, and it should begin connecting to the Deployment Server shortly afterwards.

The video below provides a demonstration of how this looks in practice.

For a more automated installation approach, see the scripted installation example located at the end of the tutorial. 

Validating the installation

Depending on the configuration of your environment, you may begin seeing logs for the host coming into Splunk shortly after the installation of the Universal Forwarder is complete and you’ve specified the deployment server. If you do, that’s generally a great indication that the deployment was successful. However, in some cases, additional troubleshooting is necessary. 

Observing App Installation and Incoming Data

Most of the time, the deployment will proceed without any issues. You can check this by looking at the etc/apps directory in the Universal Forwarder installation location and watching folders being created. These are apps that are pulled from the deployment server.

If you have access to the Web UI of the deployment server, you can check the forwarder management interface to see if the newly installed Universal Forwarder is showing up as a client. This will allow you to see the last time it checked in, as well as what serverclasses the host is a member of and what apps are installed. 

Additionally, you can search the Splunk _internal index for logs from the Universal Forwarder, as well as all indexes for the host you just installed the UF.

The following video will walk you through the process of validating your UF installation.

Validating connectivity to the deployment server

One of the most common issues we see is that the deployment server is not reachable on the network. This may be the result of a DNS issue or a firewall rule preventing connectivity on the port used, which is typically TCP/8089. 

To address this, first validate that the hostname of your deployment server resolves properly by attempting to ping the host or running a dig/nslookup command. The server may not be configured to respond to ping requests, but you should at least see the hostname resolve to an IP address. 

Next, you can use openssl’s s_client to verify that a connection can be made to the deployment server’s address/port. If you get output indicating that a certificate is returned, it means that this connection is working properly. 

openssl s_client -connect <deployment.server.name>:8089

Below is a video of both these steps.

Validate the presence of deploymentclient.conf 

Running these installation methods and configuring a deployment server will result in a configuration file called deploymentclient.conf being created somewhere on the system. While this may generally be in the etc/system/local directory of the installation location for the Universal Forwarder, it may also exist in an app. Check to make sure that this file exists and that the contents are what you expect.

The following Splunk btool command can help point you to the location of this configuration. If you get no output, it means that the configuration is missing. 

$SPLUNK_HOME/bin/splunk btool deploymentclient list --debug

Below is a video of this process.

Restarting the Splunk Forwarder Service

Sometimes the Universal Forwarder will get hung up and need to be manually restarted. If you’re a Hurricane Labs Managed Splunk Services customer, we may ask you to restart the Splunk Forwarder service if it is no longer communicating with the deployment server. Restarting the UF is often enough to resolve common issues with the forwarder, especially if it was working previously before it stopped sending data. 

The Universal Forwarder can be restarted via a few different methods:

• Via the service command: service splunk restart
• Via the Splunk executable: $SPLUNK_HOME/bin/splunk restart

Below is a video of restarting the Splunk Forwarder.

Collecting Troubleshooting Information

If all else fails, additional information may need to be collected from the system to assist with troubleshooting. This is called a Splunk Diag. See this tutorial for more information on how to collect this and send it to us for analysis (the process is the same on both Windows and Linux). 

Upgrading the Universal Forwarder

At some point, it will become necessary to upgrade the version of the Splunk Universal Forwarder on your systems. The Universal Forwarders are generally quite compatible with various versions of Splunk, but there will eventually be a time where new features are introduced or there are some breaking changes (such as improved SSL ciphers) that necessitate an upgrade. 

As a general rule of thumb, the major version of the Universal Forwarder (e.g. Splunk 8.x, Splunk 7.x) should match that of your Splunk infrastructure. It’s even better if the minor version (e.g. Splunk 8.2.x, Splunk 8.1.x) matches, but it’s fairly rare for that to be a significant issue when simply forwarding data. 

To upgrade a Splunk Universal Forwarder (assuming the tgz installation method):

• Stop the Splunk Universal Forwarder
• Untar the tgz of the new Universal Forwarder package over the existing one
• Start the Splunk Universal Forwarder
• Review the changes and continue the installation 
• Verify that the UF is still functioning properly (checking into the deployment server and forwarding data)

Below is a video showing the upgrade of a Universal Forwarder from Splunk 8.1.2 to Splunk 8.2.3.

Scripting the Universal Forwarder Installation 

If you have more than a couple Linux hosts where you need to deploy the Universal Forwarder, you’ll most likely want to use some centralized deployment mechanism (such as puppet or ansible) to deploy the Universal Forwarders. Alternatively, you may also want to use a script to deploy the Universal Forwarder to make the process quicker.

As a starting point, here’s a script that can be used to help automate the deployment of UFs in your environment. I’ve extensively tested this on a single system in my lab, so consider it to be just enough to get started with customizing the installation process to your needs. You’ll want to replace the variables in the script with values that are appropriate for your environment. 

Once this script is run, it will install the Universal Forwarder, set a admin password, configure the deployment server, start the UF service, and configure the UF to run on boot.

Below is a video demonstration of how to use this script to deploy the UF in a test environment.

Removing the Splunk Universal Forwarder

If you need to uninstall the Splunk Universal Forwarder, follow these steps:

• Disable boot start: /opt/splunkforwarder/bin/splunk disable boot-start
• Stop the Splunk process: /opt/splunkforwarder/bin/splunk stop
• Make sure no Splunk processes are still running: ps ax | grep splunk
• Delete the /opt/splunkfowarder directory (if the tgz install method was used) or use the appropriate dpkg/rpm commands to uninstall the package

Below is a video demonstration of the uninstall process on my test machine.

Note: If you’re looking to upgrade the Splunk Universal Forwarder, don’t uninstall the current one–doing so may result in duplication of data into your Splunk environment. 

Conclusion

Hopefully, this guide helps you as you deploy your Splunk environment and collect data from your Linux hosts systems. Also, be sure to check out the first part of this tutorial if you have Windows hosts that also need the Universal Forwarder installed. If you have any questions about Universal Forwarder (or Splunk) deployment best practices, reach out to us!