Have you ever been trying to use Splunk to help solve a problem, only to find out that the logs you need stopped coming into Splunk? After quite a bit of feedback from clients asking “where’s my data?”, we decided to tackle this problem. After a few years of internal development and refining, we released the Broken Hosts App for Splunk onto Splunkbase.
The Broken Hosts App for Splunk is a useful tool for monitoring data going into Splunk. It can alert you when hosts stop sending data into Splunk. It also looks at the last time that data was received by Splunk for each combination of host, sourcetype, and index. If data from that host, sourcetype, and index combination is arriving later than expected, it will send an alert so that the issue can be resolved.
This is the first part in a multi-part series. In the rest of the series, we’ll dive into the nitty-gritty details about how to set it up and do the initial configuration, as well as discuss how to respond when it alerts you to an issue.
App components overview
There are three main components that work together to make the Broken Hosts App for Splunk work. The first is four search macros that set up some base settings. The second is the “expectedTime” lookup table that contains any tuning settings that are needed. The last is a saved search, which uses the macros, the lookup table, and data from the Splunk indexes to determine if an alert needs to be generated.