How to Use Splunk to Pull Results from Elasticsearch

Alright, ready for a little brutal truth? Sometimes people don’t store their logs or other data in Splunk. There, I said it, it’s out there now. Personally, I think these sorts of people are a little off the beaten path, but hey, not everyone uses my preferred brand of deodorant either so it’s okay. Anyway, even though we all prefer Splunk, sometimes you have to deal with something big data-y that just isn’t Splunk. Enter ElasticSearch.

More truth-telling, I actually really like ElasticSearch and we’ve used it for quite a few things around here where Splunk just isn’t the right thing. Prime example, we were moving ticketing systems and wanted to keep all of our ticket history somewhere. So we spun up an ElasticSearch instance in AWS and off we went. Sometimes though, you STILL need to search the data in ElasticSearch and wouldn’t it be great to be able to do that with Splunk so you can see all your data together? Of course it would.

I set out thinking “hmm, an external search command would be the right thing.” Then suddenly, I realized, “wow, there are way smarter people than me, let’s see if someone else did this,” and as it turns out a few people have. Basically, the external search command reaches out to the ElasticSearch box and queries it based on what you send it. It’s really pretty neat.

I know what you’re thinking to yourself: “This bald guy is crazy! No way this just works!” Well, it does and here’s how to go about it:

1. cd $SPLUNK_HOME/etc/apps
2. git clone https://github.com/brunotm/elasticsplunk.git
3. Restart Splunk ($SPLUNK_HOME/bin/splunk restart
4. Enjoy!

Obviously, you’ll need an ElasticSearch installation with some data and the ability to query it, which is beyond the scope of this post. However, once you have that, run this in your Search and Reporting app in Splunk:

This should return a listing of your indexes in ElasticSearch, like so:

Now you’re able to query ElasticSearch and the data will appear as it would from any other source. I told you it was pretty neat.

That’s about it, have fun!