Splunkbuilders Association: Making Splunk ES Into Your Dream Home

There are many ways to relate the process of building a new home to the engagement of the Enterprise Security (ES) application written for Splunk. For those interested in successfully utilizing ES for their environment, I hope to convey what I’ve learned while working through ES from the ground up and piece together some analogy-style reference points for working with this product in a ‘cozier’ way.

The Foundation.

It’s important to have a firm grasp on the foundation of your log environment to equip yourself for pouring strong walls, so-to-speak, to support your security visibility. Prior to installing ES, or any Splunk-written application, you have to understand your data. This is also significantly more important with ES, as it’s not built for a specific log, external product, or data type.

This foundation is referred to as the Common Information Model (CIM), when discussed within Splunk and it’s documentation. The Common Information Model (CIM) is used to build field names that are commonly used within Splunk apps, but aren’t necessarily named as such by the vendors who have built add-ons for indexing and categorizing log data. With the CIM, you’ll have to consider every log type that gets sent into Splunk.

With anything in information security, extreme attention to detail is important. This concept is not only important for Splunk in general, but is an essential requirement for moving forward with the “framework” of your new home on ES street.

The Framework.

In working with ES, I have discovered that the framework is the most eye-opening aspect when it comes to useful security intelligence. As noted before, the framework relies heavily upon Splunk’s CIM and, with that concept, it can be placed onto your splunked data set.

This framework within ES is listed as “data models”. Data models are just that, a search logic that surrounds big picture ideas. An example model is “Malware”. With this data model, you can simply start a search across all data considered “malware” or an “attack”. I list those in quotes, as they are what’s called a data tag within splunk.

Tags help the data models identify what log source it will reference when specified in a search. To the point, if you wanted to research “malware” events in your environment, you can simply search, “index=* tag=malware”. This is one of my favorite methods of real-time “give me the raw logs” security analysis. Now, ES will not know where to put it in this upfront profiling for data modeling, so you will have to put in the upfront work for each data model and all of your indexed data.

Each of these data models build the frames and walls that allow your home to segment and separate the scenarios of daily living. They form the different “Domains” within the product; Access, Network, Endpoint, Threat and Identity. These are leveraged by built-in searches, also called correlation searches, which are the out-of-the-box product that drives ES.

There you have it. The bare minimum to having ES work properly. From here on out, ES will be searching your data and populating a new index of data (index=notable) with assumed security events. Similar to the bare minimum for living in a house – you have the foundation and framed wall structure. Sure, there may not be any furniture or a beautiful family sitting with their dog by the fire, but you won’t get wet and your family and belongings (or data) will feel safe from predators.

This bare essentials home vision may be how some people feel when they start working with the events generated by the minimum effort ES installation. Be weary though, everything sounds like a threat in a new home as you’re beginning to settle in. Although there are lots of sounds to worry about in a newly built home, don’t overwhelm yourself with every little noise. There’s still plenty more work you can do to reinforce the beautiful new home you are the proud owner of.

The Living Room.

There is a main room built within ES that you will spend the majority of your time in. Luckily, this home design comes with a fully furnished living room, meaning you have a sharable dashboard working space to live in, named “Incident Response”.

I call this your “living room” because it’s essentially where ES has a built-in ticketing system meant to be occupied by a dedicated security person or team. Within each security event is the ability to write custom comments, prioritize events, set the status of progress, and designate analyst or engineer ownership. In this “room”, you’ll be able to review every alert that ES has detected within your log data on a real-time basis. The correlation searches, governed by the data models within, will generate security events that will find their way to your living room.

The more time you spend in your living room with your notable events, the more you may find yourself wanting to make changes. After all, you do own the data that necessitates a house and you know best where it belongs and how to use it to your liking. That’s okay, as ES has the ability to customize everything from the ground up. Although, be forewarned, I would strongly recommend against customizing the built-in searches. You can always clone them and modify a similar one without worrying about the changes being lost with future upgrades.

Customizations.

Part of the fun of building a new home is making it yours. I prefer our clients to do the same with their ES environment. The longer our SOC team lives within those walls, the more changes and improvements we like to make for all involved on the investigation side of things.

ES is built to protect your data, and again, your data is special to your world. The easiest example I can make, that can seriously increase productivity of investigating your events, is this:

Add the CIM field “file_name” to your malware events in the Endpoint domain. From Symantec to Trend to McAfee to Vendor-name, this field along with many others can bring you pertinent information where it matters – right in your face. However, ES will not make the assumption that this field exists, so you have to verify what it does with your data and that it is properly being read as such (see the CIM and data modeling above).

One of the fun things to do when I show people what I believe are the inner-workings of ES is that. Simply search “index=notable” and you will have the event and fields used front and center in a format that is similar to the S&R app. This is also a great way to validate any customizations you make to the searches.

Not everyone is keen on the default pink kitchen countertops that were in the model home, but this can be addressed once you move in. As long as you keep this in mind, you can modify and adjust almost anything in Splunk itself – ES included.

Managing Contractors.

“Contractors” (like us over here at Hurricane Labs) are meant to work for you. We are your employees for the time invested. Believe me when I say that you can exhaust our abilities with this product for your end goal, which would hopefully be leading to meaningful and actionable alerts. I prefer this myself, and I’m one of those contractors. I may not be covered in sawdust or cement, but I enjoy being knee deep in your logs finding the true meaning of security events out of the alerts generated.

If you want something more than what you’re getting, by all means please tell us. I love getting on the line and working through this process and I’m certainly not alone in this. Our team has the same feeling of interest and intrigue when we work with our customers.

Everything is so much better when it makes clear sense, we believe that and we want our customers to have a solid level of understanding as well. Use us for that knowledge.

A Solid and Steady Roof.

Beyond the foundation, a roof is the most important aspect of owning a house. It takes a sturdy foundation and solid framework to support a roof. With ES, the “roof” is your events and reports. It’s the protective umbrella over your head. You know it’s there, you know what it took to build and support it.

Yet, you only worry about it when things start to go wrong. These times are expected within ES. No single security product is 100% bulletproof. Your events will change as your domain/network evolves and grows. You will need to invest the same amounts of effort every “20 years” or so after the roof has weathered the storms and changes over time.

Just remember, that it can easily be mended to patch the holes, replace the shingles, and block the leaks that allow infections to enter your home. It’s the shield above your head to the storms around you, but you have to put in a little effort to make sure it stays that way. ES is capable of the same strength for your environment. Please, don’t leave it unchecked for 20 years.

Home Sweet Home.

Once the work has been put into the ES product beyond the initial installation, I believe you’re headed to a much better “home” for organized security intelligence. Maybe I just invented a new three letter, OSI, organized security intelligence. It’s a fairly old concept to me with IDS, IPS, and other security products. It is absolutely achievable within the structures of ES – a product “tuning” process, for ES.

Once you start to get the hang of the inner workings of ES and Splunk, you’ll start to want to migrate more alerts to the workflow and incident response format. It’s a sturdy home. It’s easy to grow into. Not just for you, the lone gunman with the pure security mindset, but also to grow the mindset and delegate responsibility to others so that you can rest assured triage will be effectively engaged.

P.S. Splunk does an amazing job with documentation, so I’ll refer you there for the detailed Q/A – unless you’re a customer of ours then, of course, come talk to us.