Splunk Tutorial: CRUDing a KV Store in Splunk Using Python

By |Published On: September 14th, 2016|

In a previous blog series, I covered how to create and CRUD a KV Store using Splunk’s SPL (Search Processing Language). Feel free to check out the various pieces and parts of that tutorial here: Creating and CRUDing a KV Store in Splunk: Part 1 and Part 2.

In this blog post I’m going to cover how to do the same thing using Python. The blog post is available, as well as the subsequent screencasts. So, feel free to read, watch or do both.

In order to follow along with this you should download the Splunk Python SDK.

Export the PYTHONPATH

You can place the Splunk SDK folder where you want, but you will need to add the folder to your Python path in order to run the examples:

Copy to Clipboard

Using .splunkrc

In this example, I’m going to use a .splunkrc file to store my credentials. The .splunkrc file is a handy way for us to store our credentials when we connect to Splunk through our Python script.

You don’t have to use a .splunkrc file, but its easier than having to write this every time we want to execute a file, see below:

Copy to Clipboard

Example .splunkrc

Copy to Clipboard

Where to put .splunkrc

The location of the .splunkrc file will depend on whether or not you’re following along using Windows.

On Windows, you will put the .splunkrc in C:\Users\currenusername\.splunkrc

If you are on Linux or OSX place it in ~/.splunkrc

How does this work?

When we run our Python file, Splunk is going to check and see if user credentials have been passed into the command line. If not, it will then check if a .splunkrc file exists.

Inside of your Splunk SDK folder, there is an examples folder. This is where we will add our task_collection.py and add the following:

Copy to Clipboard

Importing connect will allow us to actually connect to Splunk. We will pass our credentials to it shortly. Next, we will try to import parse, which will be used to pull out our credentials from command line arguments or the .splunkrc file. If it cannot be imported that means you did not successfully export the SDK to your PYTHONPATH, as described above.

Copy to Clipboard

We then define the main() function where we will set up an opts variable from which we will pull out our user credentials. We will also set the owner to nobody and the app context to search. We will then connect to Splunk using our opts.kwargs.

Copy to Clipboard

Next, in the main() function, we will set up our collection name and set the collection using service.kvstore.

Copy to Clipboard

We will also check if the collection exists and if it does not, then we will create it:

Copy to Clipboard

You can then read the data from KV Store collection, using the query() function:

Copy to Clipboard

Finally, you will want to run the following, in order to execute the main() function, when you run the Python script from the command line:

Copy to Clipboard

Once you’ve added everything, save the file.

Create

To create, or insert, new data in your collection add the following to the Python script inside of the main() function above where we are printing out the collection data using the query() function:

Copy to Clipboard

All we are doing is inserting JSON to add a new value to KV Store.

Save the script and then run it on the command line. You should get back something like this:

Copy to Clipboard

Read

As described above, we can use query() to read the data from our KV Store. In our example we are specifically running this (see below) to print out the results:

Copy to Clipboard

Update

Currently, we don’t have an update in our file. Go ahead and comment out the collection.data.insert that currently exists and then add:

Copy to Clipboard

The key to updating is to first provide a string version of the _key you want to update and then as the second parameter include all the fields you want to update from your KV Store.

Save your file and rerun it, and you should see the updated values in the output.

Delete

We can either delete a specific row based on the _key:

Copy to Clipboard

Or we can delete the entire collection:

Copy to Clipboard

So, what can we do with this information? We could CRUD our KV Store from outside of Splunk using the Python SDK, or we could create a custom REST endpoint, modify our data and the CRUD the collection in some way.

In Closing

With all of this information, you should have a pretty solid understand of how to create a new KV store collection, as well as a lookup definition that allows us to communicate with our collection through the Splunk query language. We also covered how to CRUD our KV store collection through the Splunk query language as well as doing the same through Python. If you have any questions, feel free to leave them below in the comments section.

About Hurricane Labs

Hurricane Labs is a dynamic Managed Services Provider that unlocks the potential of Splunk and security for diverse enterprises across the United States. With a dedicated, Splunk-focused team and an emphasis on humanity and collaboration, we provide the skills, resources, and results to help make our customers’ lives easier.

For more information, visit www.hurricanelabs.com and follow us on Twitter @hurricanelabs.