Learn How To Build a Real-Time Filtering Table in Splunk: Part 1

By |Published On: September 22nd, 2014|

It’s pretty simple to create a table in Splunk. By default, Splunk needs to refetch the data in order to filter it down. However, what if you had a set of data and you wanted to easily filter that table in real-time?

Let’s say you have a predefined list of subnets in a lookup. You shouldn’t have to refetch the data to find a match, if you’re searching for something specific, especially since the data isn’t changing frequently enough. In this case, having something that filters in real-time would be much more effective.

I am going to take you through step-by-step how to do just that. Due to the amount of content we will be covering, this tutorial will be split into two separate posts. The first portion will cover the basics of setting up an app through the Splunk Web Framework, which will result in the creation of a custom input field and table. The second will cover how to add the filtering functionality to what we have built in the first.

Oh, and if you enjoy a more visual route, there are related screencasts split across three videos.

Already familiar with the Splunk Web Framework? You will probably be alright skimming through this first part.

Caution: There’s some heavy coding ahead, specifically in regards to JavaScript. I will do my best to guide you through each step.

Part 1: The Necessities

Download the zipped db_exploits.csv file. This contains a list of database exploits (http://www.exploit-db.com/) and we will use this data to populate our lookup. Once you have this downloaded, go into Splunk and create a new lookup table from this .csv file. We’ll be referencing it in our search as| inputlookup db_exploits.csv

Feel free to also download working examples of the app:

Part 2: Create Your App

Since we are using the built-in Splunk Web Framework, we are going to create our app from the command line at $SPLUNK_HOME/etc/apps/framework and run: ./splunkdj createapp <appname> #name whatever you like

It will then ask for your username and password and then prompt you with: The <appname> app was created at ‘$SPLUNK_HOME/etc/apps/<appname>’. Please restart Splunk.

Once you restart, go to http://localhost:8000/dj/en-us/<appname>/home/ and you should see something like this:

Default app view

Note: If you don’t restart Splunk you will get a 404 error.

In the console go back to$SPLUNK_HOME/etc/apps/<appname>/ and you will see a directory structure like this:

App name directory

Everything we will be doing will be happening inside the Django directory. Go to django/<appname>/ and you will see a directory structure like this:

Django directory

The three important directories we will be dealing with are:

  • static – this is where we keep our .css and .js files
  • templates – where Django templates are kept
  • templatetags – here we will define two new Django template tags filtertable and filterinput to be used in our Django template

First, let’s take a look at the default Django template inside of the templates directory called home.html:

Copy to Clipboard

Here’s an outline of what each section in this template is for:

  • {% extends “splunkdj:base_with_account_bar.html” %} loads the the top Splunk bar.
  • {% load splunkmvc %} loads the default Splunk mvc javascript file
  • {% block title %}{{app_name}} Home Page{% endblock title %} provides the title of the page, which would end up being ‘mynewapp Home Page’ after being rendered
  • {% block css %} is where we put either inline CSS or link to external CSS files
  • {% block content %} is where the content goes – our custom template tags that we will create shortly will go here
  • {% block js %} is where we will keep our JavaScript template

Part 3: Add the Search Manager

Before continuing, we’re going to add in a new block called {% block managers %}. This is where we will be keeping our search manager, which will call our Database Exploit lookup and populate our table.

Right after the {% endblock content %} add:

Copy to Clipboard

We provide an id of ‘dbe’ in order to reference this search later, when we add our table template tag. This is so it knows where to pull it’s data from. We are using a lookup called db_exploits.csv to populate our search and the search itself is pretty straightforward. I’m also limiting this to 500 results, because if we try to filter a ridiculous amount it could return 10,000 results and be way too performance heavy on the browser.

Part 4: Add the JavaScript template

Let’s first create our input field JavaScript template. This will be a simple input field, so there’s no need to use one of Splunk’s built in form elements. Also, because I want the input field to pass it’s value to the table, I will be using a Backbone View that will utilize this template. If you’ve never used Backbone before, don’t worry, it should all make sense once you see how it fits together. Just know that Splunk Web Framework’s JavaScript components use Backbone as their core, so it makes sense for us to do the same. For now, we will need a template to reference, which will be added into our{% block js %}. This template will be referenced inside our filterinput.js file, and will allow us to attach the functionality we define there to the JavaScript template we define in our home.html file.

Go ahead and remove the default tags inside the {% block js %} inside the Django template located at appname>/django/<appname>/templates/ and add the following and save the file:

Copy to Clipboard

Part 5: Create Custom Template Tags

At this point, we need to create custom template tags for our table and input. Go to <appname>/django/<appname>/templatetags/ and create two new files called filterinput.py and filtertable.py.

Go into filtertable.py and add the following:

Copy to Clipboard

Here we are creating our filtertable tag. The name itself is derived from the method name. If we called this method foobizbaz then in our .html template we would define it as

{% foobizbaz %}. The path to the javascript file links to the location of <appname>/django/<appname>/static, which can be confusing since it’s just <appname>/filtertable. Keep in mind that directory is where the file actually exists.

Add the following to the filterinput.py file:

Copy to Clipboard

Part 6: Add your custom template tags into the template

Go back to the home.html file located in <appname>/django/<appname>/templates/and first remove the following from the{% block content %} :

Copy to Clipboard

then add inside the {% block content %}:

Copy to Clipboard

The managerid in the filtertable tag references the search manager we added earlier, so it knows where to pull in the data from. At this point, it’s not going to know how to render these tags and, if we were to visit our page at http://localhost:8000/dj/en-us/mynewapp/home/, we would get a Django template error. This is because Django has no idea how to render these tags and this is where the JavaScript will come into play. When we use one of our custom tags, JavaScript will handle how that data should be rendered.

Part 7: JavaScript all the things

Go to <appname>/django/<appname>/static/<appname>/ and create two new .js files called filterinput.js and filtertable.js.

The basic structure of filtertable.js is as follows:

Copy to Clipboard

Looking through filtertable.js, we are first using the define() method to define a new module. Then, we load in the necessary files including Underscoresplunkjs mvc and the SimpleSplunkView.

The filter table extends the SimpleSplunkView inheriting all of its properties and providing us an easy way to handle the data that Splunk gives us from our search.

The options() method tells Splunk to return “results,” instead of the other option of a “preview.” The formatData() method is where we will eventually be formatting the data into a table format. createView() creates the view, we won’t be doing anything else with this method. updateView() updates when the table needs to be re-rendered by Splunk. Finally, we return FilterTable at the end.

As for filterinput.js, we are going to add the following:

Copy to Clipboard

There are some differences between filterinput.js file and filtertable.js file. As we won’t need to be handling data sent by Splunk, it would be unnecessary to extend from SimpleSplunkView. Instead, I am using a Backbone View, which is what SimpleSplunkView also extends from. However, we don’t need all the added Splunk specific functionality. Due to the fact we are using a Backbone View we need to load in Backbone directly up top. We are also adding a reference to our filtertable.js, since we will be connecting the two.

First, el: ‘#filterinput’ defined a pre-exisitng element in our html to attach this element to. If you remember #filterinput is defined in our template in the content block as {% filterinput id=”filterinput” %}. In theinitialize() method I am defining the template for this input, which is the JavaScript template that was added to our home.html file earlier. It then calls the render() method and attaches the template to the DOM with .

Now, before we can view this in the browser, a reference needs to be added to the JavaScript files in our Django template.

Part 8: Add a reference to your JavaScript files

Go back to your home.html template in >span class=”theme:twilight font:consolas font-size:16 line-height:25 lang:default highlight:0 decode:true crayon-inline”><appname>/django/<appname>/templates/ and right below {% load splunkmvc %} in home.html you will want to add the following:

Copy to Clipboard

If you go to view the page in the browser, you should see one line of data output below the input field:

Filter table

This is because in the updateView() method inside of filtertable.js we have var myResults = data[0] effectively pulling out the first line. What we need to do next is loop through our data, output all of the rows, and format them into an actual table.

Add the following to the formatData() method in filtertable.js method (be sure to remove the original return data):

Copy to Clipboard

Above, we first define a new empty string myDataString, followed by the Underscore method _.each() to loop through each row of data and wrap table rows and columns around them. At the end, we take the rows and wrap a <table /> tag around them so it formats it nicely.

Also, in the updateView() method, replace what is currently there with since the data has already been formatted in the >span class=”theme:twilight font:consolas font-size:16 line-height:25 lang:default decode:true crayon-inline “>formatData() method, there is nothing else we need to do here.

In the end, your filtertable.js file should look like this:

Copy to Clipboard

Now, save the file. If you go back to the browser and view the page (http://localhost:8000/dj/en-us/<appname>/home/), you should see the input field and the table. At this point, if you type anything into the input field, it won’t filter the table. This will be handled in the next part as we connect the input field to the table so it does its job and filters as we type.

About Hurricane Labs

Hurricane Labs is a dynamic Managed Services Provider that unlocks the potential of Splunk and security for diverse enterprises across the United States. With a dedicated, Splunk-focused team and an emphasis on humanity and collaboration, we provide the skills, resources, and results to help make our customers’ lives easier.

For more information, visit www.hurricanelabs.com and follow us on Twitter @hurricanelabs.