Don’t Sink Your Security Ship by Making These SIEM Mistakes

By |Published On: August 6th, 2015|Tags: , |

Time to Fortify Our Senses and Steer Clear of the SIEM Sharks

I’ve been adventuring in the SIEMs for some time now, learning what I can and striving to be successful across both management and consultative realms. Over the course of my career, I have a few bruises and horror stories from my trials and errors, but I have many victories to show for them as well. Throughout my experience, I have found some common mistakes connected to SIEMs.

Although it would be great if it were true, SIEM is not a one-stop shop for all things security. It is not a device that can be plugged in and will automatically be the watchers on the wall, looking for nefarious activity. There are a lot of aspects that need to be understood to achieve SIEM success, including cost, manpower, dedication, and more.

Be Wary, SIEM Investment Goes Beyond the Sticker Price

Understanding the investment of what you’ve actually purchased is the first step to having a successful monitoring program. SIEM is an enticing subject, one that even has luck getting company buy-in from the higher-ups… until they see the price tag that is. Too many times there’s the perception that a SIEM can simply be bought and that the money spent will go far enough on its own to provide security.

Any SIEM is going to cost money, just like any appliance is going to cost money. Unfortunately, that dollar amount doesn’t include the investment in manpower that coincides with properly standing up and running a SIEM. A SIEM has to be properly configured, instructed and tuned. This is not a plug-and-play, one-size-fits-all solution. Without a lot of devotion a SIEM can be a pretty price-heavy paperweight.

The Amount of Manpower Needed is Often Misunderstood

One of the major aspects that is too often overlooked is the manpower necessary for a SIEM. We live in a day and age of machine-learning, fire-and-forget, plug-and-play devices. Why should our enterprise solutions be any different? Because, for anything worthwhile, there is some assembly required.

Let’s take a look at this from different angles. From the internal perspective it’s much more complex than simply looking at SIEM-generate alerts. Everything from making sure the initial configuration is done properly, to having the ops team configure data sources correctly, is important. On the consultant side, the work is in providing an understanding of the investment, ensuring confidence that they can do the legwork (since most companies have a thing against giving a third party domain admin), and sharing knowledge about the length of time the entire process takes, are all keys to a successful client relationship.

The Good Ol’ Silence Is Golden Rule Doesn’t Ring True Here

I’m going to derail things for a moment to delve a bit into the nature of the client relationship when it comes to managed SIEM services. One of the mistakes that I have both seen and been guilty of myself is being too quiet.

Picture this: The program is going nicely, the rules are tuned, alerts are quickly investigated, filed away, and so forth. The client will only be hearing from you if something is a serious issue. Out of sight, out of mind. Idyllic, right?

Wrong. And this can transform itself into a problem very quickly. If your client only hears from you on the rare occasion something bad is happening, they start to wonder what they paid for. Questions start to arise. How are we spending so much money on somebody that only responds to a devices rules on occasion? Does their team do any work beyond plugging the appliance in? How often do they look at the device?

Silence is very definitely your enemy in this instance. Being too quiet can actually give the misimpression that, now that the monitoring program has hit a balance of activity, that it must be an easy task. In my professional career, I have witnessed the responsibility over an SIEM program, one that hasn’t yet achieved a good standard, put in the hands of a new intern. This occurred because I left them with the impression that the heavy lifting was done and over with, when that wasn’t necessarily the case at all.

The fact is communication is key. Discussions about what tasks have been accomplished, what still needs to be done, plans about the future, etc. Good communication is tantamount to technical knowledge for managing a client’s SIEM environment.

Please, Try Not To Annoy Your Help Desk Allies

People invest in SIEM technology mainly for the alerting and monitoring aspects, right? That’s not too surprising. Alerting is one of, if not the, most important function with a SIEM. If configured properly, it can provide actionable information towards security and operational issues. When configured improperly, however, it will annoy everybody that has to deal with it.

Attempting to follow up on every alert coming from a misconfigured SIEM, will cause you to end up chasing a lot of false positives. Eventually, the help desk staff that goes to investigate the endpoints and should be one of your greatest allies, will stop answering your calls and start to avoid eye contact with you, you’ve goofed up somewhere. Having a standard for investigation, something to confirm malicious activity, before contacting the help desk to track down the offending machine will help avoid this issue.

Increasing Versatility Can Lessen the Chance for Error

Another issue is forgetting to test rules using realistic scenarios. A problem that testing goes through is that it will test the exact scenario that the alert was built for, but not test the versatility of the rule. Engaging a pentest team or local hackerspace can help build appropriate threat models for testing how thoroughly the rules have been generated. That versatility is key to have accurate rules that test for various situations, rather than a large list of rules that only target specific ones.

Loading the SIEM is also an area that I have seen a lot of errors on. In my opinion, it does no good to put every data source that your organization has as a mistake. It causes there to be a flood of information, allowing useful information to potentially fall through the cracks. What should be started, at least initially, is for there to be a diversity of data sources (firewalls, domain controllers, mail servers, etc), instead of a large quantity of a single source. Once those data sources are properly organized and tuned, adding large quantities of data sources shouldn’t matter, so long as you follow the same convention.

Word To The Wise

Following these observations will hopefully help you create a monitoring program that maximizes the use of a SIEM application, whether you are on the consultant side or internal looking to strengthen your company’s security posture. Avoiding common pitfalls is a fast way to quick wins that can further those goals.

About Hurricane Labs

Hurricane Labs is a dynamic Managed Services Provider that unlocks the potential of Splunk and security for diverse enterprises across the United States. With a dedicated, Splunk-focused team and an emphasis on humanity and collaboration, we provide the skills, resources, and results to help make our customers’ lives easier.

For more information, visit and follow us on Twitter @hurricanelabs.