Deploying the Splunk Universal Forwarder on Windows
The Splunk Universal Forwarder is the best mechanism for collecting logs from servers and end-user systems. In order to collect logs at scale, it is necessary to deploy the Universal Forwarder to every system where log collection is required. Managing the deployment of the Universal Forwarder is best handled via whatever mechanism your organization uses to deploy software packages across machines in your organization. However, if you’re doing a one-off installation of the Universal Forwarder or don’t have a method of deploying MSIs, the installer may be an acceptable option.
In this tutorial, we’ll explore how to deploy the Splunk Universal Forwarder on a Windows machine using the MSI package provided by Splunk.
If you’re interested in learning how to install the Universal Forwarder on Linux, click here!
Obtain the Installation Package
First, download the Splunk Universal Forwarder from Splunk’s download page. You will need a Splunk.com account to access the download. In the event you need to download an older version of the Universal Forwarder, those packages are available on the older releases page.
For this process, you’ll want to download the MSI package for your version of Windows.
When downloading a Universal Forwarder, pay attention to the versions of Windows that are supported by the package. For example, newer versions of the Universal Forwarder, such as 8.1.x, don’t support older versions of Windows server, such as Windows Server 2012 or Windows Server 2012 R2.
If you’re a Hurricane Labs Managed Splunk Services customer, our support team can advise you on what packages are best suited for your environment and provide the MSI if you don’t have a Splunk account available.
Gather Required Information
When installing this, there are two options: one is using the MSI with arguments, and the other is using the GUI installer. In order to proceed with either option, you’ll want to first have the following information:
- Deployment Server: This is the host in your Splunk environment that manages configuration on all of your universal forwarders. This should be a DNS CNAME whenever possible to make future updates or server migrations easier. We do not recommend specifying the IP address of a deployment server when applying this configuration.
- Username and password: This should be a unique username and password that will be configured on the Universal Forwarder and used in the event of any configuration changes or troubleshooting needed in the future. In versions of Splunk preceding 7.1, this was automatically set to admin/changeme, but this is now a required parameter due to security concerns around a default password.
Using the MSI Installation
For most clients, using the MSI installer with arguments makes the most sense. You can do the deployment via the MSI with some configuration flags. The installation arguments for the MSI are detailed in the Splunk documentation.
You’ll want to use the following arguments on this install:
With all of these set the /quiet flag should also work.
For example, the following msiexec command would install the universal forwarder to connect to the deployment server of ccnproddeploy01.customerscallnow.com and set a username and password.
Below is a video that will walk you through the process of running this installation.
Using the GUI Installer
When using the GUI installer, there is an option for specifying the deployment server, which, as in the above example, should be a DNS CNAME or the hostname of your Splunk system. You will also be asked to set a username/password, which is something to keep on hand in case you need to run something locally on that system for troubleshooting purposes. All of the other settings (including inputs and data forwarding) get handled once the system connects to the deployment server.
When running the installation wizard, you will be asked if you’re deploying the Universal Forwarder for an on-premise or Splunk Cloud deployment. If you have an environment managed by Hurricane Labs with a deployment server, you’ll always want to choose the on-premise option (even if you’re a Splunk Cloud customer), since all of the configurations will be managed by the deployment server.
One of the options in the installer will allow you to specify the deployment server. Once again, you’ll want to use a DNS CNAME or hostname for this setting, and typically the port will almost always be 8089.
The video below will walk you through the process of running this installation.
Validating the installation
Depending on the configuration of your environment, you may begin seeing logs for the host coming into Splunk shortly after the installation of the Universal Forwarder. If you do, that’s generally a great indication that the deployment was successful. However, in some cases, additional troubleshooting is necessary.
Observing App Installation and Incoming Data
Most of the time, the deployment will proceed without any issues. You can check this by looking at the etc/apps directory in the Universal Forwarder installation location, and watching folders being created. These are apps that are pulled from the deployment server.
Additionally, you can search the Splunk _internal index for logs from the Universal Forwarder, as well as all indexes for the host you just installed the UF.
The video below will walk you through the process of validating your UF installation.
Validating connectivity to the deployment server
One of the most common issues we see is that the deployment server is not reachable on the network. This may be the result of a DNS issue or a firewall rule preventing connectivity on the port used, which is typically TCP/8089.
To address this, first validate that the hostname of your deployment server resolves properly by attempting to ping the host. The server may not be configured to respond to ping requests, but you should at least see the hostname resolve to an IP address.
Next, if the system has a web browser, you can actually navigate to the address of the deployment server in a web browser, using the format https://deploymentserver:8089. If you see something like the screenshot below, it means that this connection is working properly.
Below is a demo for both these steps.
Validate the presence of deploymentclient.conf
Running the installation via either the MSI or GUI method and specifying the deployment server for the UF to use will result in a configuration file called deploymentclient.conf being created in the etc/system/local directory of the installation location for the Universal Forwarder (generally C:\Program Files\SplunkUniversalForwarder). Check to make sure that this file exists and that the contents are what you expect.
Below is a video of this process.
Identifying the deploymentclient.conf location
Depending on how your Universal Forwarder was deployed, it may not be immediately obvious where the deploymentclient.conf file is located. This can certainly be a challenge when your environment has been around for a while, or there have been significant configuration changes.
Fortunately, Splunk provides a command line tool that can be used to identify the configuration that is being used by the forwarder. Let’s use this tool, called btool, to identify where the configuration is located.
First, open an administrative command prompt on your Windows system. I find the easiest way to do this is to search for “cmd” in the Start menu, right click on it, and choose “run as administrator”.
Note: if you follow these steps and get an “access is denied” message as output, you’re most likely not using an administrative command prompt.
Next, you’ll want to run the following command:
Note: you may need to adjust the path based on where your Universal Forwarder is installed.
The output of this command will show you the location of the deploymentclient.conf configuration files on this host. In this example, you’ll see that the deploymentclient.conf file is in the all_deploymentclient app.
I’ve put together a demonstration of this process, which you can view below.
Restarting the Splunk Forwarder Service
Sometimes the Universal Forwarder will get hung up and need to be manually restarted. If you’re a Hurricane Labs Managed Splunk Services customer, we may ask you to restart the Splunk Forwarder service if it is no longer communicating with the deployment server. Restarting the UF is often enough to resolve common issues with the forwarder, especially if it was working previously before it stopped sending data.
Below is a demo of restarting the Splunk forwarder.
Collecting Troubleshooting Information
If all else fails, additional information may need to be collected from the system to assist with troubleshooting. This is called a Splunk Diag. See this tutorial for more information on how to collect this and send it to us for analysis.
Hopefully, this guide helps you as you deploy your Splunk environment and collect data from more systems. If you have any questions about Universal Forwarder (or Splunk) deployment best practices, reach out to us!
About Hurricane Labs
Hurricane Labs is a dynamic Managed Services Provider that unlocks the potential of Splunk and security for diverse enterprises across the United States. With a dedicated, Splunk-focused team and an emphasis on humanity and collaboration, we provide the skills, resources, and results to help make our customers’ lives easier.
For more information, visit www.hurricanelabs.com and follow us on Twitter @hurricanelabs.