How to Customize Splunk Alerting with Webhooks

One of the best features of Splunk is the ability to configure a nearly endless amount of actions based on your log data. This tutorial will guide you through a simple example of using a Webhook action in Splunk to notify users upon a successful authentication to a system.

The environment we’ll be using is the one that’s built out in my Splunk course, “Getting to Know Splunk: The Hands-On Administration Guide”. If you haven’t taken this and want to learn more about how to get started with Splunk, I’d highly recommend checking it out.

Screencast

For those who prefer a more audio-visual style of learning, I’ve created a screencast that walks you through this process as well!

The Scenario

Suppose you have a server you manage and you know you should be the only user with any SSH access to the machine. Therefore, you want to be notified of any successful logins to this machine. Also, since you believe in living dangerously, you don’t have a host firewall configured on this system. We won’t question your motives, but assume you have a good reason to do this.

Prerequisites

A Splunk environment containing the data you want to alert on:

• For this example, we’ll be using the Ubuntu authentication log at /var/log/auth.log.

An available Webhook to catch your alert:

• A webook is a mechanism to allow one app to send information to another in real-time. This allows data (in this case, from Splunk) to be pushed to some other application as soon as a relevant search or alert finishes running.
• For this example, I’ll be using Zapier (https://zapier.com/).

Let’s Get Started

Splunk makes this process pretty easy. Here’s how you do it:

1.  Verify that your data is in Splunk, and write a search that will serve as the trigger to your alert.

In this example, the search below would cover what we’re looking for:

2.  Adjust the search to run over an appropriate timeframe. In this example, we’ll run the alert every 5 minutes, and set up the search to look over the last 6 minutes of data. This will account for any possible delays in receiving the data.

This is what our search looks like:

3.  Save the search as an alert:

Note that to run every 5 minutes, you will want to use a cron expression. If you’re not familiar with cron syntax, https://crontab.guru/ is a great resource.

4.  Next, add a trigger action to your alert. For this example, we’ll use the webhook option.

5.  Now we need a Webhook. In Zapier, click “Make a Zap!”, and choose “Webhook by Zapier” as the trigger.

6.  Choose the option to “Catch Hook”:

7.  Zapier will give you a custom Webhook URL to use for this action. Copy this to the clipboard, and add it to Splunk.

(Note that if you’re using a training/dev installation without a license, this will no longer continue to function after the trial period expires).

8.  Ensure that a sample event is generated and that your alert will run. Once this is completed, click “Ok, I did this” to view the sample alert data in Zapier. You will see the data from your Splunk alert displayed.

9.  Now you need an action step. Zapier supports tons of options, but for the sake of this example, we’ll just send an email.

10.  We will choose Gmail to send an email:

11.  You can be creative here using fields from the Splunk alert to generate a customized message:

12.  View the test message, and send it to your email if desired.

13.  Name your Zap, and turn it on. Now you’ll get a notification within about 5 minutes of any successful access to your Linux server.

The Possibilities Are Endless

While this is just a basic example (which could be accomplished using much simpler mechanisms), consider this more of a prototype than something you might actually use as-is. The real flexibility in here is the ability to use the Webhook for nearly any other application. Tools such as Zapier add even more flexibility by chaining actions together based on the same hook.

A more complex option might be something like this:

• Don’t just use email for notification: open a ticket in Zendesk, while also notifying a Slack room and sending a text message. Format each message in a way that’s appropriate for the medium, all with one notification from Splunk. For example, the ticket may contain very detailed information, where the SMS message just provides enough information to notify an on-call engineer.

This becomes more flexible for a use case that’s a bit more complex than just detecting logins. Consider a e-commerce example:

• Web logs detect an error in the checkout process
• Splunk alert is generated, webhook is received
• Ticket is created to notify the development team of the issue
• Customer support is notified of the issue so they can reach out to the customer

The possibilities are pretty endless as to what you can accomplish. Using an external tool with a Webhook really provides an extra degree of flexibility and customization beyond what the out-of-box alerting allows, enabling you to use the data from Splunk to power and automate many different types of processes.