Threat Hunting with Splunk: Part 2, Process Creation Log Analysis

In the previous part of this series, we introduced you to process creation log sources in Windows, relevant data fields for analysis, and instructions on how to import this data into Splunk. This post is going to focus on some basic queries you can use to interrogate those logs and how to filter benign results. There will also be a small section on hypothesis and questions you can ask of this data to help discover anomalous activity.

Queries to Jump Start your Threat Hunting

There are a wide variety of possibilities and all sorts of different ways you can manipulate the data depending on what it is you are trying to do. I’m going to show you a couple of introductory searches I put together that might give you some ideas on where to start, and then give you some ideas for manipulating the data in the form of questions you could use process creation logs to answer.

This is as good a starting point as any. You’ll probably get a large number of responses back. A lot of threat hunting is starting with broad queries and getting more and more specific as you have more and more questions or things you want to filter out.

This search queries the “WinEventLog” sourcetype (substitute this with the sourcetype you are dumping your windows event logs to). We’re looking for all EventCode 4688 entries (process creation). From there, we’re piping this query to the stats command, listing out all of the executables that have been seen for a given time period, and sorting them by how many times they have been executed, AND by similar command line arguments.

Results can then be sorted by executables that are run least frequently or most frequently (all you have to do is click the “count” field to sort by most or least frequent occurrences). These results can be investigated through other queries in order to build context and/or determine intent. Here is a similar query using sysmon logs:

Just like the Windows Process logs, expect a large number of events back. We’ll get into looking at specific processes and/or filtering in just a moment.

Here is an example WinEventLog query, specifically looking for powershell.exe process creation events:

You’ll notice the first few entries are being run by Splunk. Everything else though? That probably warrants a closer look. Look at those command line arguments.

Here’s something identical for sysmon logs:

Unfortunately, not quite as exciting as the other query I ran with a sample dataset, but it still gets the point across.

As many are already aware, Powershell is a scripting language created by Microsoft, built with system administrators in mind. A lot of newer Microsoft software is built with integration into Powershell. Microsoft wants system administrators to use Powershell for managing and maintaining Windows networks. Of course, while the intended users were system administrators, security researchers began to take a closer look at the scripting language. As a result, several tools were developed with the red team in mind. Eventually, bad guys discovered the advantage of using Powershell for payload delivery, persistence, lateral movement, etc.

So, this query is a good start and all, but what can I do if I run into things I don’t want to see and/or want to filter out? Try this WinEventLog query:

This query builds on our initial query, with some key differences. First, we’re telling Splunk that we want to look at the entries in the New_Process_Name field that end with powershell.exe:

Next, we’re using the clause:

to filter out some results.

Let’s say you have an endpoint product that runs a lot of Powershell scripts and calls powershell.exe to execute them repeatedly, and you want to ignore those entries. The above clause says “Do not show me any log entries that have any of these keywords/phrases in the “Process_Command_Line” field.

As an example, let’s say your security product is called TotesSecure and runs Powershell scripts out of C:\Program Files\TotesSecure\Tool Scripts\, and that there are a lot of scripts here. Here is how you would filter it out:

We entered the string “*TotesSecure\\Tool\ Scripts*” in the parentheses. The wildcard character (*) at the beginning and end of the string allows us to to pattern match without having to know the full directory path for the “TotesSecure” Tool Scripts directory. The wildcard at the end of our string also means that we don’t have to filter every single script in the Tool Scripts directory; we want to filter on all of them. You may also notice that I used a backslash (\) character to escape both the space in the “Tool Scripts” directory name and the backslash in the TotesSecure directory path.

You can use this filtering technique to filter more than one keyword or string in a single query. For example, let’s say you have Legit Monitoring Agent installed, and you’re sick of seeing it repeatedly in your Splunk results while zeroing in on unusual powershell.exe executions, but you also want to filter out the “TotesSecure” Tool Scripts directory. Simply add a comma for each keyword/string you want to filter on:

The rest of our query is simply piping our query to stats and counting results based on both the New_Process_Name field (which should always return powershell.exe in our example), and the Process_Command_Line field (which will show us all the unique arguments passed to the powershell.exe executable).

The vast majority of your activity hunting through process logs is being able to filter suspected noise out entirely, or for later investigation on its own.

Just to make sure that the sysmon users are included, here is a similar query that performs more or less the same functions:

This query is almost identical to the WinEventLog query. All you’re doing is swapping out field names where necessary.

What if you want to search for more than one executable in New_Process_Name/Image, but still want to keep the filters you set up via Process_Command_Line/CommandLine field? Well, here is how I did it for WinEventLog:

Instead of using New_Process_Name=”*blah.exe”, we use the clause

Just like in previous queries, we can use wildcards in order to not have to match the full directory path to the executable (e.g. *cmd.exe), and we can use commas to tell our query that we want to match on more than one New_Process_Name entry (e.g. *cscript.exe, *powershell.exe).

This works almost the same as what we did for filtering directory paths/phrases out of the CommandLine/Process_Command_Line fields for the previous queries.

And just for consistency, here is a similar query for sysmon users that works identically:

That’s really all there is to it. All of the queries I’ve used for process creation log hunting are some variation of these two initial queries:

“Show me all of the EventID 4688 or Sysmon EventID 1 events for a given time period, and sort them by least frequent occurrence”

“Use the list of executables from the first query, and focus in on a single (or smaller number of) executables, show me the arguments that got passed to the executable, and sort them by the least frequent command line arguments”.

An Important Warning about Filtering Data

As demonstrated in the queries above, Splunk makes it pretty easy to filter out benign results. However, It’d be irresponsible if I didn’t mention that some advanced adversaries can and will take advantage of overly permissive filters in order to operate in the margins. A lot of advanced adversaries will execute tools out of directories containing antivirus tools, or rename their tools to make them appear to be legitimate.

With this in mind, you have two options. We’re going to pick on “Legit Monitoring Agent” to demonstrate these options with examples:

Option 1: Use an overly broad filter to filter out all results containing the string “Legit Monitoring Agent”. Then, once your investigation is complete, formulate a new query focusing exclusively on results from “Legit Monitoring Agent”. Let’s start with both WinEventLog and Sysmon examples:

These queries are going to return all results with process names that include “Legit Monitoring Agent”. This filter will work for examining binaries that run from that directory, but if the “Legit Monitoring Agent” uses cscript or powershell to run scripts for various purposes, you may need to use the Process_Command_Line/CommandLine field instead. Here are queries for WinEventLog and Sysmon, respectively:

Or, if you really wanna be crazy, use the OR operator to cover executables and/or scripts execution paths containing “Legit Monitoring Agent” in a single query. Once again, here are queries for WinEventLog and Sysmon:

Option 2: Let’s consider ways we can make our filter less broad. The original filter string was

*Legit\ Monitoring\ Agent*

What is the directory structure for this software (e.g. C:\Program Files (x86), C:\Program Files, etc.)?

Is there a subdirectory containing scripts (e.g. C:\Program Files (x86)\Legit Monitoring Agent\Scripts) and/or executables (e.g. C:\Program Files (x86)\Legit Monitoring Agent\bin)?

Are we looking to filter all of a particular file extension (e.g. .ps1, .psm, .psd1, .exe files, etc.)?

Or are we looking to filter just one file at a time?

These questions can help you make a fine-tuned filter. For this example, let’s assume we want to filter C:\Program Files\Legit Monitoring Agent\bin\lmagent.exe and C:\Program Files\Legit Monitoring Agent\scripts\networkinfo.ps1. I’m going to show you a couple of queries (for both WinEventLog and Sysmon) that you can use to tune your filtering, going from less specific to more specific:

These queries will return process creation logs for everything, except new processes executing out of either Program Files\Legit Monitoring Agent\bin or Program Files (x86)\Legit Monitoring Agent\bin, and any processes that have a command line that references the Legit Monitoring Agent\scripts directory (again either Program Files OR Program Files (x86)). Let’s take this a step further:

These queries operate almost identically to the ones above, the only difference is that we added “*.exe” to the New_Process_Name/Image to ignore all executables in the Legit Monitoring Agent\bin directory, and “*.ps*” to ignore all .ps1, .psd, or .psm powershell files in the Legit Monitoring Agent\scripts directory. Now, lets try this one more time:

Here, we specifically filter out lmagent.exe and networkinfo.ps1. A much more precise filter than where we started. Remember that you can use commas to add more values (e.g. more directories and/or specific files) to ignore in our queries. If you end up hunting through process creation logs a lot and are required to filter out a lot of files and directories, you may want to become familiar with utilizing lookup tables and importing a csv file filled with all of the directories and files you would like to ignore.

Use Cases and Questions to Drive Process Creation Log Hunting

I promised some use cases and/or questions you can ask your process creation event logs to help drive your threat hunting investigations. Here are some things to think about:

• If you have sysmon logs onboarded into Splunk and you have the ‘Hashes’ field extracted, try using a search like:

This is a sysmon-specific search that will show you a list of executables observed over a time period, including their file hash, and give you a count of how many times that executable name and file hash combination has been seen. You can then extract those file hashes and feed them to threat intelligence tools, or, for example, search for the file hash on VirusTotal (either via the web GUI or via their API) to see if the file(s) have been seen before and/or whether or not different endpoint security products see the file as malicious or not.

• What if all I have are Windows native tools and pre-installed third party applications in my process creation logs?

Have you ever heard of the concept of “Living off the land?” Well, attackers and adversaries are getting smarter, and learning to use Windows native tools and applications for everything from initial compromise, pivoting, persistence, and everything else in between. The LOLBAS project is helping defenders enumerate the various ways that Windows native tools can be abused by adversaries. If the LOLBAS project isn’t enough for you, consider taking a look at the MITRE ATT&CK framework. A lot of the techniques described by the Mitre framework utilize OS-native tools, and have references to real-world threat groups and malware campaigns where these techniques have been observed in the wild.

Between MITRE ATT&CK and LOLBAS, there is plenty of information out there you can use to determine what constitutes unusual activity for a native application. Pay attention to the Process_Command_Line/CommandLine fields to spot unusual arguments to common Windows binaries. While we’re talking about Windows Native tools and patterns of unusual activity, think about some of these things as well:

• Are these commands being executed at an unusual time (e.g. outside of business hours and/or normal working hours for the given user)?
• Have the commands being executed on a given system ever been executed on that system before (e.g. do you really think that users/systems in the accounting group would normally be seen running netstat.exe, net.exe or whoami.exe)?
• Much like the previous question, Has the Security Id/Account Name/User been seen executing particular system commands before? Why all of a sudden are they running system commands now? Are service accounts suddenly running commands that they have NEVER run before?
• Are the processes being logged executing out of unusual directories?

Most legitimate installed applications are going to be running out of %WINDIR% (e.g. C:\Windows, normally) %WINDIR%\System32, %WINDIR%\SysWOW64, Program Files (x86), or Program Files. Of course, there are exceptions because sometimes application developers just hate us all, and certain applications will just spew executables everywhere and execute out of user home directories and all sorts of weird places.t However, in some cases, things that are NOT executing out of one of the five directories mentioned above should probably be scrutinized a little bit further. Pay attention to the full path in the New_Process_Name/Image field and/or the Process_Command_Line/CommandLine.

This small collection of use cases for process creation logs is far from complete, but I hope it gives you some inspiration and ideas for how you can turn mountains of logs into actionable data.

Conclusion Coming Soon

In this blog post we continued to build on the material covered in Part 1, focusing on queries you can use in Splunk to interrogate your process creation logs. We also took a bit of time to describe different strategies you can use to filter out benign results from your queries in order to reduce the volume of data analysts have to sift through. Finally, we discussed some hypotheses and specific questions you can use to guide hunting through your process creation logs.

In the final post of this series, We’re going to go over a hands-on example together–I’m going to guide you through investigating results in a sample dataset that you can follow along with, using the information and guidance you have acquired so far. Additionally, I will provide links to additional reading material and resources to help guide your threat hunting. Stay tuned!