Splunk Hands On: Deployment Servers and Server Classes
Getting data into our Splunk system
Most Splunk instances we deal with here at Hurricane Labs are what we call “distributed environments”. This means that there are multiple Splunk instances running as a fleet to serve a greater good. When an environment matures enough to require more than just one server, you will want to make sure you have centralized management from the beginning – trust me, figuring this out before you make the mistake of slapping local configs everywhere is a solid win.
Whether or not you’re using an all-in-one Splunk server (one Splunk instance doing the indexing, searching, and anything else you want to make Splunk do), or you are using a distributed environment, we can all agree on something:
You want all the data, from all the sources… or at least some data, from some sources.
So, how do you maintain configurations on multiple endpoints to send data to your environment? What happens when you have hundreds of clients and have to change one line of code because you added another indexer? Centralized management for Splunk can extend further than just the Splunk Enterprise infrastructure itself.
This is where the concept of Universal Forwarders (UFs) comes into play. These tiny, lightweight versions of Splunk can be installed on almost all recent operating systems and can be configured remotely from a Splunk Deployment Server (DS) once installed. These UFs are configured to act as Deployment Clients to the deployment server via settings on the client in a file called deploymentclient.conf. While there are many items that can be configured between a deployment server and client, we are going to focus on deploying deployment-apps to clients to get data into our Splunk system.
Note: I have also created a video (below) to go along with this blog post for the visual learners of the world. So, feel free to read through the post, watch the screencast, or both!
How does the deployment server work?
Your deployment server interface can be found under Settings > Forwarder Management.
A deployment server’s job is to glue together deployment-apps with deployment clients. Deployment-apps are regular apps/TAs that sit in $SPLUNK_HOME/etc/deployment-apps/ on your DS. When a client checks in, if it matches a server class (defined by hostname or IP), it will download the apps associated to that server class. A server class is the glue that maps deployment-apps to clients and is configured in serverclass.conf on the deployment server.
Time to test out a server class
Let’s walk you through making a test server class for some visual learning.
Once I’m on my Forwarder Management page, I’ll click “New Server Class”. I’ll choose to call it “test_windows_servers” as I want to bring in Windows event log data. Once this is created, I can start adding deployment-apps to my server class.
When adding apps, all available apps in $SPLUNK_HOME/etc/deployment-apps will show under “unselected apps”. Filtering through these apps, I can find the apps I want to deploy and simply click them to add them to the server class before saving the configuration.
Now that you have apps assigned to a server class, you’ll want to add map client machines by clicking “add clients” on the server class’s screen. From here, we can specifically whitelist or blacklist machines by hostname, DNS name, client name, instance name, IP address, or even machine type.
I’m going to filter by only Windows clients since I’m attempting to pull events from the event log from my domain controllers.
As you can see, I’m applying a wildcard to the servers I want, because my domain controller hostnames all start with “ccndc0”. I am also filtering on windows-x64 machines as a precaution to not deploy to Linux servers if I butcher my whitelist config.
Clicking “preview” will allow you to see which hosts you’ll match prior to saving the configuration. Once saved, you can view all details of the server class.
Once configured, your client will check, in as shown above, and download the configurations (deployment apps).
You have now remotely deployed a configuration to a Splunk client, which can be extremely useful when working with a large organization where you have hundreds of deployment clients.
Happy Splunking!
About Hurricane Labs
Hurricane Labs is a dynamic Managed Services Provider that unlocks the potential of Splunk and security for diverse enterprises across the United States. With a dedicated, Splunk-focused team and an emphasis on humanity and collaboration, we provide the skills, resources, and results to help make our customers’ lives easier.
For more information, visit www.hurricanelabs.com and follow us on Twitter @hurricanelabs.
