Part 1: Set Up the KV Store and Lookup Definition
KV Stores, unlike regular Splunk Lookup files, contain a unique ID field called _key. It’s a much more stable way to update your data than doing so through a regular Lookup file. KV Stores are also much more performant than lookup files. Lookup files must be replicated across indexers, which can cause performance problems if they become too large. KV Stores on the other hand, do not need to be replicated across indexers. This is also built on MongoDB, so it may make more sense to those experienced with working with databases versus dealing with a lookup file, which is more of a flat file that doesn’t contain any sort of unique attribute in it (_key).
Prerequisites: You should know how to navigate through Splunk’s UI (user interface). You should also have a basic understanding of Splunk’s query language, as well as how lookups work.
Note: We have both blog and screencast pieces of this tutorial available. So, if you would prefer a more audiovisual option feel free to view the screencasts.
What are we going to cover?
We’re going to cover how to create a KV Store both through the UI, as well as by modifying collections.conf and transforms.conf. KV Stores in splunk are nothing more than Mongo Databases, so they allow us to easily apply CRUD (Create / Read / Update / Delete) to our data.
We will also cover editing a KV Store directly using the Splunk Search Language.
Build the KV Store
I highly recommend using the Splunk Lookup Editor to create and edit your lookup files and KV Store collections. I will first show you how to create the KV Store collection using the old-fashioned way of creating a collections.conf file. I will then show you how to create it using the Splunk Lookup Editor, as well as through curl on the command line.
Option 1: KV Store and Lookup Definition Creation Through .conf Modification
KV Store Creation Using collections.conf
Go into the Search app’s local directory and create a collections.conf file. Inside of collections.conf add the following:
Save the file.
Lookup Definition Creation Using transforms.conf
Navigate to the Search app’s local folder and create a transforms.conf file, if one doesn’t already exist.
In the transforms.conf file add the following:
Now, there are a couple of ways we can try to refresh transforms.conf. The documentation says we can try running | extract reload=t in the search window. As an admin we can also try running the following in the browser:
If all else fails, restart Splunk.
Option 2: KV Store and Lookup Definition Creation Through Splunk’s UI
KV Store Collection Creation Through the UI
If you’ve installed the Splunk Lookup Editor, navigate to that app and create a new KV Store: