Threat Intelligence Aggregation and Deduplication with MineMeld

Recently, I’ve been getting familiar with an open-source project by Palo Alto Networks called MineMeld. The best way I can find to describe MineMeld is that it’s almost like an RSS feed reader for threat intelligence feeds. You configure MineMeld to acquire threat intelligence data through its “miner” functionality to go out and mine threat intel. MineMeld has numerous miners for acquiring threat intel from a wide variety of sources. You then use a processor to determine what type of indicators are going to be processed from the mined data. For instance, IPv4 addresses, URLs, user-agents, file hashes (md5, sha-256, ssdeep, etc.), filenames, mutexes, registry keys, and so on. Finally, the processed indicators can be linked to an output mechanism for consumption by analysts and/or security products downstream. The illustration below from the MineMeld web interface shows miners in blue, processors in red, and outputs in yellow, with lines illustration how data flows through the MineMeld instance.

What is meant by downstream consumption? Well, for some examples, you might take a collection of IP addresses and add them to your firewall. You might produce of a list of malicious URLs or domains and feed them to your proxy to block malicious websites faster. You might feed IP addresses, domains, and/or user-agents into a database and have your analysts utilize it for cross referencing IDS alerts or other security incidents to see if the indicators you gathered can help determine if an event is a true positive requiring further investigation, or a false positive. This graphic from Palo Alto Networks may help describe where and how minemeld fits in with your enterprise’s security.

The documentation I wrote guides users through the process of acquiring MineMeld, setting a virtual machine, installing MineMeld, guiding users through the MineMeld interface, modifying existing miners to collect threat intel from data sources MineMeld does not support by default, and much more. The guide can be found in PDF form here.