That’s really all there is to it. All of the queries I’ve used for process creation log hunting are some variation of these two initial queries:
“Show me all of the EventID 4688 or Sysmon EventID 1 events for a given time period, and sort them by least frequent occurrence”
“Use the list of executables from the first query, and focus in on a single (or smaller number of) executables, show me the arguments that got passed to the executable, and sort them by the least frequent command line arguments”.
An Important Warning about Filtering Data
As demonstrated in the queries above, Splunk makes it pretty easy to filter out benign results. However, It’d be irresponsible if I didn’t mention that some advanced adversaries can and will take advantage of overly permissive filters in order to operate in the margins. A lot of advanced adversaries will execute tools out of directories containing antivirus tools, or rename their tools to make them appear to be legitimate.
With this in mind, you have two options. We’re going to pick on “Legit Monitoring Agent” to demonstrate these options with examples:
Option 1: Use an overly broad filter to filter out all results containing the string “Legit Monitoring Agent”. Then, once your investigation is complete, formulate a new query focusing exclusively on results from “Legit Monitoring Agent”. Let’s start with both WinEventLog and Sysmon examples:
These queries are going to return all results with process names that include “Legit Monitoring Agent”. This filter will work for examining binaries that run from that directory, but if the “Legit Monitoring Agent” uses cscript or powershell to run scripts for various purposes, you may need to use the Process_Command_Line/CommandLine field instead. Here are queries for WinEventLog and Sysmon, respectively:
Or, if you really wanna be crazy, use the OR operator to cover executables and/or scripts execution paths containing “Legit Monitoring Agent” in a single query. Once again, here are queries for WinEventLog and Sysmon:
Option 2: Let’s consider ways we can make our filter less broad. The original filter string was
*Legit\ Monitoring\ Agent*
What is the directory structure for this software (e.g. C:\Program Files (x86), C:\Program Files, etc.)?
Is there a subdirectory containing scripts (e.g. C:\Program Files (x86)\Legit Monitoring Agent\Scripts) and/or executables (e.g. C:\Program Files (x86)\Legit Monitoring Agent\bin)?
Are we looking to filter all of a particular file extension (e.g. .ps1, .psm, .psd1, .exe files, etc.)?
Or are we looking to filter just one file at a time?
These questions can help you make a fine-tuned filter. For this example, let’s assume we want to filter C:\Program Files\Legit Monitoring Agent\bin\lmagent.exe and C:\Program Files\Legit Monitoring Agent\scripts\networkinfo.ps1. I’m going to show you a couple of queries (for both WinEventLog and Sysmon) that you can use to tune your filtering, going from less specific to more specific:
These queries will return process creation logs for everything, except new processes executing out of either Program Files\Legit Monitoring Agent\bin or Program Files (x86)\Legit Monitoring Agent\bin, and any processes that have a command line that references the Legit Monitoring Agent\scripts directory (again either Program Files OR Program Files (x86)). Let’s take this a step further:
These queries operate almost identically to the ones above, the only difference is that we added “*.exe” to the New_Process_Name/Image to ignore all executables in the Legit Monitoring Agent\bin directory, and “*.ps*” to ignore all .ps1, .psd, or .psm powershell files in the Legit Monitoring Agent\scripts directory. Now, lets try this one more time:
Here, we specifically filter out lmagent.exe and networkinfo.ps1. A much more precise filter than where we started. Remember that you can use commas to add more values (e.g. more directories and/or specific files) to ignore in our queries. If you end up hunting through process creation logs a lot and are required to filter out a lot of files and directories, you may want to become familiar with utilizing lookup tables and importing a csv file filled with all of the directories and files you would like to ignore.
Use Cases and Questions to Drive Process Creation Log Hunting
I promised some use cases and/or questions you can ask your process creation event logs to help drive your threat hunting investigations. Here are some things to think about:
- If you have sysmon logs onboarded into Splunk and you have the ‘Hashes’ field extracted, try using a search like:
This is a sysmon-specific search that will show you a list of executables observed over a time period, including their file hash, and give you a count of how many times that executable name and file hash combination has been seen. You can then extract those file hashes and feed them to threat intelligence tools, or, for example, search for the file hash on VirusTotal (either via the web GUI or via their API) to see if the file(s) have been seen before and/or whether or not different endpoint security products see the file as malicious or not.
- What if all I have are Windows native tools and pre-installed third party applications in my process creation logs?
Have you ever heard of the concept of “Living off the land?” Well, attackers and adversaries are getting smarter, and learning to use Windows native tools and applications for everything from initial compromise, pivoting, persistence, and everything else in between. The LOLBAS project is helping defenders enumerate the various ways that Windows native tools can be abused by adversaries. If the LOLBAS project isn’t enough for you, consider taking a look at the MITRE ATT&CK framework. A lot of the techniques described by the Mitre framework utilize OS-native tools, and have references to real-world threat groups and malware campaigns where these techniques have been observed in the wild.
Between MITRE ATT&CK and LOLBAS, there is plenty of information out there you can use to determine what constitutes unusual activity for a native application. Pay attention to the Process_Command_Line/CommandLine fields to spot unusual arguments to common Windows binaries. While we’re talking about Windows Native tools and patterns of unusual activity, think about some of these things as well:
- Are these commands being executed at an unusual time (e.g. outside of business hours and/or normal working hours for the given user)?
- Have the commands being executed on a given system ever been executed on that system before (e.g. do you really think that users/systems in the accounting group would normally be seen running netstat.exe, net.exe or whoami.exe)?
- Much like the previous question, Has the Security Id/Account Name/User been seen executing particular system commands before? Why all of a sudden are they running system commands now? Are service accounts suddenly running commands that they have NEVER run before?
- Are the processes being logged executing out of unusual directories?
Most legitimate installed applications are going to be running out of %WINDIR% (e.g. C:\Windows, normally) %WINDIR%\System32, %WINDIR%\SysWOW64, Program Files (x86), or Program Files. Of course, there are exceptions because sometimes application developers just hate us all, and certain applications will just spew executables everywhere and execute out of user home directories and all sorts of weird places.t However, in some cases, things that are NOT executing out of one of the five directories mentioned above should probably be scrutinized a little bit further. Pay attention to the full path in the New_Process_Name/Image field and/or the Process_Command_Line/CommandLine.
This small collection of use cases for process creation logs is far from complete, but I hope it gives you some inspiration and ideas for how you can turn mountains of logs into actionable data.
Conclusion Coming Soon
In this blog post we continued to build on the material covered in Part 1, focusing on queries you can use in Splunk to interrogate your process creation logs. We also took a bit of time to describe different strategies you can use to filter out benign results from your queries in order to reduce the volume of data analysts have to sift through. Finally, we discussed some hypotheses and specific questions you can use to guide hunting through your process creation logs.
In the final post of this series, We’re going to go over a hands-on example together–I’m going to guide you through investigating results in a sample dataset that you can follow along with, using the information and guidance you have acquired so far. Additionally, I will provide links to additional reading material and resources to help guide your threat hunting. Stay tuned!