Deploying the Splunk Universal Forwarder on Windows

When downloading a Universal Forwarder, pay attention to the versions of Windows that are supported by the package. For example, newer versions of the Universal Forwarder, such as 8.1.x, don’t support older versions of Windows server, such as Windows Server 2012 or Windows Server 2012 R2. 

If you’re a Hurricane Labs Managed Splunk Services customer, our support team can advise you on what packages are best suited for your environment and provide the MSI if you don’t have a Splunk account available. 

Gather Required Information

When installing this, there are two options: one is using the MSI with arguments, and the other is using the GUI installer. In order to proceed with either option, you’ll want to first have the following information:

• Deployment Server: This is the host in your Splunk environment that manages configuration on all of your universal forwarders. This should be a DNS CNAME whenever possible to make future updates or server migrations easier. We do not recommend specifying the IP address of a deployment server when applying this configuration. 
• Username and password: This should be a unique username and password that will be configured on the Universal Forwarder and used in the event of any configuration changes or troubleshooting needed in the future. In versions of Splunk preceding 7.1, this was automatically set to admin/changeme, but this is now a required parameter due to security concerns around a default password. 

Using the MSI Installation

For most clients, using the MSI installer with arguments makes the most sense. You can do the deployment via the MSI with some configuration flags. The installation arguments for the MSI are detailed in the Splunk documentation. 

You’ll want to use the following arguments on this install:

With all of these set the /quiet flag should also work.

For example, the following msiexec command would install the universal forwarder to connect to the deployment server of ccnproddeploy01.customerscallnow.com and set a username and password.

Below is a video that will walk you through the process of running this installation.

Using the GUI Installer

When using the GUI installer, there is an option for specifying the deployment server, which, as in the above example, should be a DNS CNAME or the hostname of your Splunk system. You will also be asked to set a username/password, which is something to keep on hand in case you need to run something locally on that system for troubleshooting purposes. All of the other settings (including inputs and data forwarding) get handled once the system connects to the deployment server. 

When running the installation wizard, you will be asked if you’re deploying the Universal Forwarder for an on-premise or Splunk Cloud deployment. If you have an environment managed by Hurricane Labs with a deployment server, you’ll always want to choose the on-premise option (even if you’re a Splunk Cloud customer), since all of the configurations will be managed by the deployment server.

One of the options in the installer will allow you to specify the deployment server. Once again, you’ll want to use a DNS CNAME or hostname for this setting, and typically the port will almost always be 8089.

The video below will walk you through the process of running this installation.

Validating the installation

Depending on the configuration of your environment, you may begin seeing logs for the host coming into Splunk shortly after the installation of the Universal Forwarder. If you do, that’s generally a great indication that the deployment was successful. However, in some cases, additional troubleshooting is necessary. 

Observing App Installation and Incoming Data

Most of the time, the deployment will proceed without any issues. You can check this by looking at the etc/apps directory in the Universal Forwarder installation location, and watching folders being created. These are apps that are pulled from the deployment server.

Additionally, you can search the Splunk _internal index for logs from the Universal Forwarder, as well as all indexes for the host you just installed the UF.

The video below will walk you through the process of validating your UF installation.

Validating connectivity to the deployment server

One of the most common issues we see is that the deployment server is not reachable on the network. This may be the result of a DNS issue or a firewall rule preventing connectivity on the port used, which is typically TCP/8089. 

To address this, first validate that the hostname of your deployment server resolves properly by attempting to ping the host. The server may not be configured to respond to ping requests, but you should at least see the hostname resolve to an IP address. 

Next, if the system has a web browser, you can actually navigate to the address of the deployment server in a web browser, using the format https://deploymentserver:8089. If you see something like the screenshot below, it means that this connection is working properly.

Below is a demo for both these steps.

Validate the presence of deploymentclient.conf 

Running the installation via either the MSI or GUI method and specifying the deployment server for the UF to use will result in a configuration file called deploymentclient.conf being created in the etc/system/local directory of the installation location for the Universal Forwarder (generally C:\Program Files\SplunkUniversalForwarder). Check to make sure that this file exists and that the contents are what you expect.

Below is a video of this process.

Identifying the deploymentclient.conf location

Depending on how your Universal Forwarder was deployed, it may not be immediately obvious where the deploymentclient.conf file is located.  This can certainly be a challenge when your environment has been around for a while, or there have been significant configuration changes.

Fortunately, Splunk provides a command line tool that can be used to identify the configuration that is being used by the forwarder.  Let’s use this tool, called btool, to identify where the configuration is located. 

First, open an administrative command prompt on your Windows system.  I find the easiest way to do this is to search for “cmd” in the Start menu, right click on it, and choose “run as administrator”.

Note: if you follow these steps and get an “access is denied” message as output, you’re most likely not using an administrative command prompt.

Next, you’ll want to run the following command:

Note: you may need to adjust the path based on where your Universal Forwarder is installed. 

The output of this command will show you the location of the deploymentclient.conf configuration files on this host.  In this example, you’ll see that the deploymentclient.conf file is in the all_deploymentclient app.

I’ve put together a demonstration of this process, which you can view below.