How to Update Your Splunk Forwarders for Proper Performance

What happens when universal forwarders fall behind?

During the life of your Splunk infrastructure, it’s common to see universal forwarders fall behind several versions. In some cases, administrators are unaware of these conditions until something happens that negatively impacts operations and the collection of logs from universal forwarders on systems not primarily tasked to run Splunk. This is an easily overlooked issue until there’s a problem which could potentially compromise the collection of data or the security of the whole system.

Let’s take a look at an example of what could happen if a universal forwarder is installed on a Windows server. While this machine contacts a deployment server to get the latest deployed Splunk apps, it can’t get an upgrade to the version of Splunk running on this machine. This could result in compatibility issues with updated deployed apps. Another very real problem I see from time to time are old forwarders not sending data because they have expired SSL certificates.

Let’s take a look at resolving this problem

While outdated universal forwarders can become a burden, the issue can be identified and resolved relatively easily.

Step 1

The first step includes identifying which hosts are running old versions. This can be accomplished by performing the following search from your search head:

Clicking on the different fields in the statistics tab can rearrange the results in order. Sorting by version will allow you to see the newest version of Splunk hosts first or the oldest first.

Step 2

After identifying outdated hosts, the next step is planning a method to upgrade hosts. In Windows environments, updates can be accomplished by downloading the correct MSI package and deploying it via GPO, or other technologies like Altiris or LANDESK. Linux/Unix systems may require manual installation if hosts are not managed by a tool like Puppet or Chef.

Universal forwarders can be downloaded from this URL: https://www.splunk.com/en_us/download/universal-forwarder.html Also, if you aren’t looking to get the latest version of Splunk universal forwarder (which may be the case in many situations), there’s a link on that page for older versions.

Watch Out for Compatibility Issues…

It’s important to pay attention to which versions of Splunk universal forwarder are compatible with your operating system/s. Updating Splunk to a version not supported by the version of Windows (or any other operating system) running will return negative results and could potentially compromise your fishbuckets. Splunk compatibility information can be found at the following link: http://docs.splunk.com/Documentation/Splunk/6.5.1/Installation/Systemrequirements.

Please note that the link above is for Splunk 6.5.1. There’s a dropdown in the top right corner where different versions of Splunk can be selected in order to compare compatibility with your operating system and hardware. In most cases, it’s a good idea to not install a universal forwarder that is newer than the version of Splunk running on your indexers, search heads, and intermediate forwarders. The search provided above will list your forwarders as well as any other host running splunk that is searchable by your search heads/search peers.

Other ways of finding the version of Splunk your infrastructure hosts are currently running include:

• Clicking on Help and selecting About from that menu.
• From the terminal, run the following command:

Note: If the splunk bin directory is not in your $PATH, you may have to run the command from the command using a full path to the binary. The below suggestions are based off of default installation locations:

In Windows:C:\Program Files\SplunkUniversalForwarder\bin\splunk version

(Note: SplunkUniversalForwarder may not be the exact naming convention used, but it will be evident which folder it is in under Program Files).

In Linux/UNIX systems:

(Note: splunkforwarder may not be the exact naming convention used, but it will be evident which folder it is in under /opt).

Step 3

The final step after deploying/installing a new version of the universal forwarder is to verify that the host is reporting to Splunk and sending data as expected. You should be able to see this host checking in to your deployment server, and searching for data from the host that was upgraded is also recommended. Something like the following search would verify windows event log data is being consumed by Splunk from the newly updated host:

Sometimes a FQDN may be required to see results from this machine. A simple suggestion would be to append a * character at the end of the host name in order to avoid confusion, save time, and return the expected results. Another good search will include searching internal logs. In most infrastructures, the ability to search internal logs is only provided to system administrators. Please ensure that you are an administrator before running the following search:

Troubleshooting

If you experience issues and are unable to get logs from hosts that were upgraded (assuming they were working before the upgrade), you may need to get a diag from the host and open a support ticket for further investigation. Diag instructions are listed below:

For Windows:

From a command line, cd to

(Note: SplunkUniversalForwarder may not be the exact naming convention used, but it will be evident which folder it is in under Program Files).

For Linux/UNIX:

(Note: splunkforwarder may not be the exact naming convention used, but it will be evident which folder it is in under /opt).

You will see output on the terminal/command prompt indicating the work that is being performed as well as the destination where it was saved. Collect this tarball from your problem host, and open a ticket with support to help you resolve the issue.

In Conclusion

Managing and maintaining your Splunk infrastructure is essential to its proper operation. If forwarders are neglected during the update/upgrade process, data loss and parsing issues could be eminent and left unnoticed until someone detects a problem. If you love your Splunk infrastructure, be sure to share that love with your forwarders too.