Stress-Testing Your SIEM: Is Your Environment Actually Catching the Bad Guys?

In today’s security landscape, we have more tools than ever (EDR, XDR, SOAR, SIEM) and very little time to learn each one fully. Every tool out there advertises “we use MITRE” and “we are a Gartner top X” and now since 2025 “We have AI!”. All of these are well and good, however they all have one major flaw: once installed, they require configuration and business tuning to be fully taken advantage of. This is true for every antivirus or EDR platform, SIEM, vulnerability scanner, and SOAR platform.

This isn’t inherently bad; it is actually by design. Businesses and organizations are incredibly diverse in their environments, tooling, and business logic and security tools must be adaptable to them. Additionally, compliance frameworks such as PCI DSS, HIPAA, and SOC2 all have different priorities and required controls that businesses need to account for. 

This introduces two questions: “Is your defensive tooling actually detecting what it is supposed to detect?” and “How do you know it’s working?”

The short answer is: “Test your detections.” 

Testing security tools can be challenging and, more importantly, time consuming. Compared to a SIEM, tooling like antivirus or EDRs are often a black box with some controls, but they often lack fine tuning or transparent controls like a SIEM does. This often leads to an adversary accounting for an EDR in the case where if they can bypass it, they may evade detection for a significant amount of time.

Testing baseline and example:

First, we must cover the elephant in the SOC: Many SIEM rules are fragile. However, this does not mean the rules are useless. 

Consider a simple detection written in Splunk Processing Language (SPL). Mimikatz is one of the best-known post-exploitation tools used in Windows environments. It is open source and has been updated many times over the years.

tstats count values(Processes.process) AS processes
FROM datamodel=Endpoint.Processes
WHERE Processes.process=”*mimikatz*”
BY Processes.dest

In simple terms, if the captured process command line contains the term `mimikatz`, the rule triggers. A copy of `mimikatz.exe` renamed `mimidogs.exe` will not trigger this as the base detection string has now been changed. 

While examples like this are simple, it shows that by simply changing the name of a process an adversary can successfully evade this more narrowly written detection logic. However this detection is not useless as even threat actors can be lazy. An excellent example of this is the breach from 2022 including Okta and their 3rd party provider,  where the threat group Lapsus$ was identified as downloading the mimikatz tool directly from Github to then execute. 

How to test a SIEM:

Testing a SIEM is conceptually quite simple: You choose a technique, and execute it in a controlled manner and tune detections and preventative controls to accommodate. Then you try to identify if the technique’s goal can be achieved by an alternate way and perform that. This is often done by penetration testers as part of a pentest, redteam, or purpleteam exercise. Offensive tooling is used in a safe and controlled manner in order to prove effectiveness and show a gap in security tooling that can then be remediated to increase defensive controls. The goal is to detect techniques, not necessarily tools directly. 

Break each test into steps. A single tool or technique may generate multiple logs and indicators of compromise (IOCs), and each stage offers another opportunity for detection. David Bianco’s Pyramid of Pain is a useful way to think about the durability of those indicators. From bottom to top, it moves through hash values, IP addresses, domain names, network and host artifacts, tools, and tactics, techniques, and procedures (TTPs).

This pyramid is a good indicator for defenders that shows how potentially fragile a detection rule could be. From an offensive perspective the inverse is also true. 

The lower levels are generally easy for defenders to use but also easy for attackers to change. Rotating malware hashes is common for adversaries now and often threat actors are able to rotate hash values within minutes if not unique hashes per victim. IP addresses also rotate quickly in cloud hosted scenarios. Domain names take a bit longer as they require registration and configuration, but domains are relatively cheap and rotating between multiple is additionally common. 

The upper levels, network and host artifacts, tools, and TTPs, usually require more effort to understand and detect, but detections built around them tend to be more durable. They focus on what an adversary is doing and how the activity appears in the environment, rather than relying only on a static signature.

Bypassing Signature Detection 

Revisiting the example from earlier, but this time in more detail from and attackers position and trying to evade some detections. 

As an attacker we can plan our attack to evade several of the more fragile detections: 

• Build the tool ourselves from source and remove any public signatures, including the name of the tool. This will prevent this from being picked up via a simple name, hash or known file IOC.
• Serve the payload or communicate from a “safe” domain and/or IP to avoid poor network reputation or known malicious URLs.

From an attacker’s perspective, these two steps alone have eliminated three of the easiest ways to detect threats. Further detection from here may also include EDR or monitoring tool bypasses in order to successfully evade tooling that specifically looks for behavioral indicators. While not discussed in detail here, Windows and EDRs provide additional tooling like ETW and kernel protections that can be bypassed to successfully evade detection. However, each step that requires detection is another opportunity for defenders to create detections around. 

Bypassing Behavioral Detections

While testing SIEM rules, this is often a place where things break down as defenders are forced to identify behavior instead of direct IOCs.

Moving to a new example on how to identify and bypass “behavioral” detections. We can move to a popular cloud solution, Microsoft EntraID (formerly Azure Active Directory) and attempt to bypass password spray detections.

Common cloud attack techniques may start with password spraying attacks where an attacker attempts to use a common password, like “SprintTime2026!”, and try to login to multiple accounts using this password. Adversaries often use prebuilt tools to perform this technique as manually logging into multiple accounts is tedious. One such tool is SprayCannon. 

An example command of this may look like the following: 

spraycannon -s o365 -p “SpringTime2026!” -u target_emails.txt

This will attempt to perform a login against EntraID using the same password for all users in `target_emails.txt`. 

As a defender this is simple to detect with the following splunk search over the last hour:

| tstats count from datamodel=Authentication where Authentication.action = “failure” by Authentication.src
| where count > 100

In short, this rule looks for all authentication failures across all users and if there is a high number of failures from 1 source it will trigger an alert. 

As an adversary, we can bypass this by simply rotating our IP address. There are multiple ways to accomplish this, however one convenient method is to use multiple containers or cloud instances to perform this. By setting up proxies across residential proxies, aws ec2 instances, digital ocean droplets, or azure vms, an adversary can drastically increase the number of IP addresses that their attack comes from effectively breaking this detection. 

Assuming the role of an adversary we can achieve this by using the tool `proxychains` which allows us to forward network traffic through proxies. Assuming an adversary set up enough proxies, the final command may look like this: 

proxychains4 -q spraycannon -s o365 -p “SpringTime2026!” -u target_emails.txt

Depending on the number of proxies, this may move a single IP attempting to log into 500 accounts to 200 different IPs attempting to log in. This reduces a single IP logging in 500 times to only 2.5 times. 

Looking back at the pyramid of pain we can apply the top three layers and look for the following: 

• TTPs – The number of logins from a single IP or user within a timeframe 
• Tools – Key indicators that are attributed to a tool, such as a unique user agent for a password spraying tool 
• Network Artifacts – User agents or headers that are unique to the attack

As an adversary we can bypass many of these using optional features of this particular tool as well. We can supply a common list of current browser user agents to appear as if we are using a browser and not a password spraying tool. Additionally, we can add a delay to each attempt. 

Spraycannon -s o365 -u users.txt -p “SpringTime2026!” –useragent browser_useragents.txt –jitter 2000 –dealy 3600

This way we only generate an attempt every two seconds and if we are using more than one password it will wait an hour before the next attempt for any given account. A low-and-slow pattern like this may defeat rules that use only short time windows and per-source thresholds. That does not make behavioral detection ineffective; it shows why behavior needs to be evaluated across multiple entities, signals, and time windows.

Where to go from here?

While this article only focuses on two examples, it shows how quickly SIEM rules can be broken and emphasizes the need for testing these rules. Creating and tuning rules to detect alternate behavior is the next step and for many organizations, this must be done individually as differences in remote work, internal policies, compliance, VPN connections and other requirements lead to variations of these rules per environment. 

Conclusion

Detection testing is extremely important for validating existing SIEM rules and the effectiveness of early detection relies on how thorough the rules are. 

A detection-testing exercise or program can start with a penetration test, or it can be followed up with a penetration test or purple team exercise. Start with one high-risk technique, establish a baseline, vary the obvious indicators, tune the control, and test again. The goal is not one perfect detection per rule, its multiple rules that may overlap to create a layered detection approach. 

Sources: 

1. Lapsus$ involved breach of Okta and 3rd party https://www.netsecurity.com/how-lapsus-breached-okta-and-its-customers-okta-lapsus-hack-explained/ 
2. Mimikatz wikipedia as to not link directly to the git page: https://en.wikipedia.org/wiki/Mimikatz 
3. Password spraying tool: https://github.com/CausticKirbyZ/SprayCannon
4. Pyramid of pain: https://detect-respond.blogspot.com/2013/03/the-pyramid-of-pain.html