Creating and CRUDing a KV Store in Splunk: Part 1
Part 1: Set Up the KV Store and Lookup Definition
KV Stores, unlike regular Splunk Lookup files, contain a unique ID field called _key. It’s a much more stable way to update your data than doing so through a regular Lookup file. KV Stores are also much more performant than lookup files. Lookup files must be replicated across indexers, which can cause performance problems if they become too large. KV Stores on the other hand, do not need to be replicated across indexers. This is also built on MongoDB, so it may make more sense to those experienced with working with databases versus dealing with a lookup file, which is more of a flat file that doesn’t contain any sort of unique attribute in it (_key).
Prerequisites: You should know how to navigate through Splunk’s UI (user interface). You should also have a basic understanding of Splunk’s query language, as well as how lookups work.
Note: We have both blog and screencast pieces of this tutorial available. So, if you would prefer a more audiovisual option feel free to view the screencasts.
What are we going to cover?
We’re going to cover how to create a KV Store both through the UI, as well as by modifying collections.conf and transforms.conf. KV Stores in splunk are nothing more than Mongo Databases, so they allow us to easily apply CRUD (Create / Read / Update / Delete) to our data.
We will also cover editing a KV Store directly using the Splunk Search Language.
Build the KV Store
I highly recommend using the Splunk Lookup Editor to create and edit your lookup files and KV Store collections. I will first show you how to create the KV Store collection using the old-fashioned way of creating a collections.conf file. I will then show you how to create it using the Splunk Lookup Editor, as well as through curl on the command line.
After we create the KV Store, we will then need to set up a corresponding lookup. First, I will show how to create the lookup through the UI, then I will cover the second option of creating a lookup tied to a KV Store by modifying transforms.conf. Having this lookup in place will allow us to use | inputlookup and | outpulookup commands, which are the two main ways we will update the data in our KV Store using Splunk’s query language. A little bit later in this series we will also look at how to edit the KV Store using JavaScript as well.
Option 1: KV Store and Lookup Definition Creation Through .conf Modification
KV Store Creation Using collections.conf
Go into the Search app’s local directory and create a collections.conf file. Inside of collections.conf add the following:
Save the file.
Lookup Definition Creation Using transforms.conf
Navigate to the Search app’s local folder and create a transforms.conf file, if one doesn’t already exist.
In the transforms.conf file add the following:
Now, there are a couple of ways we can try to refresh transforms.conf. The documentation says we can try running | extract reload=t in the search window. As an admin we can also try running the following in the browser:
http://<your_splunk_address>/en-US/debug/refresh?entity=admin/transforms-lookup
If all else fails, restart Splunk.
Option 2: KV Store and Lookup Definition Creation Through Splunk’s UI
KV Store Collection Creation Through the UI
If you’ve installed the Splunk Lookup Editor, navigate to that app and create a new KV Store:
Name the KV Store spp_collection and put it in the Search app. Add the necessary fields:
Click ‘Save Lookup’.
While we could make Estimated_Completion_Date a time field, for the sake of simplicity we will leave it as a string.
Once we save it we can easily add new values to our KV store using the Lookup Editor interface:
Above, you will notice in the right most dropdown it says, “Showing entries for: nobody”. We can view the data in our KV Store, “for: nobody” meaning it’s globally accessible or for our particular user, which we cannot access through the Splunk Query Language.
Lookup Definition Creation Through the UI
Go to Settings < Lookup then click, and then next to Lookup Definitions click ‘Add New’.
Under Supported Fields, we will add all the fields from the KV Store. We want to also make sure to add the _key field, which will act as a unique identifier for our fields.
Select the destination app. In our case, it is our Search app.
- The name should be different than the collection name. Here we have called it “task_lookup”.
- The type is KV Store.
- Define the name of the collection we created to connect this lookup to the KV Store.
- Under Supported Fields add the following comma-delimited list: _key,Task_Name,Task_Description,Estimated_Completion_Date,Notes,Status
We don’t need to worry about advanced options. Click on ‘Save’.
In Part 2, we will cover CRUDing our KV Store Collection. Follow us on Twitter @hurricanelabs for updates!
About Hurricane Labs
Hurricane Labs is a dynamic Managed Services Provider that unlocks the potential of Splunk and security for diverse enterprises across the United States. With a dedicated, Splunk-focused team and an emphasis on humanity and collaboration, we provide the skills, resources, and results to help make our customers’ lives easier.
For more information, visit www.hurricanelabs.com and follow us on Twitter @hurricanelabs.
