Bold = Where a valid, publicly-trusted SSL certificate is required
HEC Data > Load Balancer > All Splunk Indexers
- Pros: Resilient + Maximum throughput and performance + Scalable + Splunk best practice
- Cons: Harder to configure + May cost more
HEC Data > Load Balancer > Set of Splunk Indexers or Heavy Forwarders
- Pros: Resilient + Good throughput and performance + Scalable
- Cons: Harder to configure + May cost more
HEC Data > Load Balancer > Single Splunk Enterprise Server (IDX/HF)
- Pros: Scalable
- Cons: Harder to configure + Single Point of failure + May cost more
HEC Data > Single Splunk Enterprise Server (IDX/HF)
- Pros: Simple Configuration + May cost less
- Cons: Single Point of failure + May require additional config to be open to the internet
Ultimately, you need to ask yourself a few questions to figure out which design to use:
- What AWS data will I need to bring into Splunk now and in the future?
- Will I have any other HEC sources that can use the same infrastructure?
- Can I tolerate a single point of failure?
- Do I need scalability?
- What does my current Splunk environment look like and how will it change?
- Can I expose my Splunk server(s) to the internet?
The simple answer is if you can’t tolerate a single point of failure, need scalability, or have a large amount of data (all HEC sources, not just AWS), then you need a load balancer front end with either an indexer cluster or set of heavy forwarders on the backend. Something else to consider is that you will have significantly lower performance if you send to a Heavy Forwarder instead of an Indexer directly. See this Splunk Blog post: Universal or Heavy, that is the question.
We usually veer toward having a load balancer in front of an indexer cluster. This is Splunk best practice, and it’s not terribly more difficult or expensive compared to the other options. If you don’t set this up the right way the first time around, you are going to put yourself and/or your team into a situation that is difficult to fix as more data sources are added.
SSL + Internet Connection
Many AWS services, such as Kinesis Firehose, send data over the internet and require a valid SSL connection to the remote target. This means your remote target will require a valid publicly trusted certificate AND be reachable from the internet on your configured HEC or LB listening port. If you’re using a load balancer, that is what your remote target will be. If you’re sending straight into Splunk, that is what your remote target will be. Often times it’s going to be a security best practice to expose a load balancer to the internet rather than your application.
Load Balancer Prerequisites
If you’re going to be using a load balancer (AWS or otherwise) to handle the WAF data, then make sure you do the following:
- Use a Classic Load Balancer if using AWS
- Setup a valid, publicly-trusted SSL certificate on the LB
- Enable duration-based sticky sessions & set cookie expiration to disabled
- Load Balancer Protocol MUST be HTTPS, port can be whatever you’d like
- Instance Protocol and port MUST match what you configured in Splunk (HTTPS & 8088 in this demo). Note: For non-AWS load balancers, this is referring to the target port/protocol, commonly referred to as port forwarding
- You may also want to refer to the AWS Documentation for more information
Regardless of your setup, you will need to meet the following network requirements:
AWS > Load Balancer Setup
- Kinesis must be able to reach the load balancer on your configured listening port
- Load Balancer must be able to reach Splunk HEC server(s) on your configured HEC port/protocol
AWS > Splunk (direct) Setup
- Kinesis must be able to reach the Splunk HEC server on your configured HEC port/protocol
Remember, Kinesis comes in through the internet. You can not route it through a VPN or VPC to avoid that. You will either need to allow traffic from all internet sources or restrict it to just Firehose CIDR’s. I’d recommend using the CIDR’s. Feel free to check out the documentation on Firehose CIDR’s.
Something else to remember is that AWS NACL’s and some network appliances are stateless, meaning you may need to explicitly allow return traffic. AWS Security Groups and other network appliances are stateful, so you only need to explicitly allow the incoming traffic.
Also important to note, if you are using a proxy, put in an exception for this traffic so it’s not intercepted.
Splunk Configuration and Setup
Believe it or not, setting up Splunk is going to be the easiest part of this. You need to do three things:
First, Install the AWS WAF Add-on
The AWS WAF Add-on needs to be installed on your HEC receiver(s) and your Search Head(s). Installation should not require a restart, but you will have to perform a debug/refresh if it was untar’d or unzipped via the CLI. Below are the typical locations for app installations:
SH Cluster: on the deployer in $SPLUNK_HOME/etc/shcluster/apps
Indexer Cluster: on the cluster master in $SPLUNK_HOME/etc/master-apps
Anything else: in $SPLUNK_HOME/etc/apps. Note: Unless you want to use a Deployment Server, then it goes in $SPLUNK_HOME/etc/deployment-apps and the app would be added to the target’s serverclass.
Second, Enable HEC on your HEC receivers
Settings > Data Inputs > HTTP Event Collector > Global Setting
- All Tokens: Enabled
- Enable SSL: Checked
- HTTP Port Number: Use the default 8088 or a port of your choice. Note: If you’re running Splunk as non-root, you may not be able to use privileged ports such as 443.
- Use the app “splunk_httpinput” (in newer versions of Splunk you can put this in any app context) and create a local/inputs.conf in the app:
- For SSL, it will use your splunkd certificate by default (path is in serverCert under the [sslConfig] stanza in server.conf). If you need to, you can specify a different SSL certificate under the [http] stanza in inputs.conf.
Lastly, Create the AWS WAF HEC Input
Settings > Data Inputs > HTTP Event Collector > New Token
- On the first screen, type a name and enable indexer acknowledgment
- On the sourcetype dropdown select “aws:waf”
- For app context, store in “TA-aws_waf”
- For the index, I recommend starting with a “test” index, before using your production index. Be sure to come back and change this later if you do!
- Save the HEC token for setup in AWS.
- Within TA-aws_waf create a local/inputs.conf with the following config:
- Note, when working through the CLI you may need to either run a debug/refresh or restart Splunk to get the changes to become live.
AWS WAF Configuration and Setup
Moving over to AWS is where things may become a bit less straightforward. The way your AWS setup is deployed may be very, very different from mine. I’ll be sticking to the core concepts here, and you can adapt them to fit your environment.
Kinesis Firehose Setup
1. Go into Kinesis and click “Create Delivery Stream”
2. Name the stream “aws-waf-logs-splunk”. Note: The start of your stream name needs to match what is in bold, or this will NOT work.
3. Skip to “Step 3: Choose a destination.” From here, click Splunk, enter your remote endpoint with the listening port you setup, and Enter the token that you created in Splunk earlier. Select (or create) an S3 bucket that will store events that fail to send.
4. Click Next to “Step 4: Configure settings.” At the very bottom under permissions, create or choose a new IAM role. When this is done, click allow.
5. Finalize and create the delivery stream
Testing Kinesis Firehose
1. Once your new stream is active from the previous step, click it.
2. Click “Start sending demo data.”
3. Wait a couple of minutes then search “index=test” in Splunk.
4. Once you see an event, click “Stop sending demo data” in AWS. Note: If you do not have data in Splunk at this point, something is not setup correctly. If you click the “Splunk Logs” tab in your Kinesis Delivery Stream, you may get some hints. Otherwise you will need to troubleshoot– see the Splunk and Amazon troubleshooting documentation – likely starting at the network layer.
5. Back in the Splunk, change the index from “test” back to your production index. This can be done by following the same CLI/GUI steps from earlier.
Enabling WAF Logging
1. In AWS go over to the “WAF & Shield” service, and find the Web ACL(s) you want to enable logging on.
2. Click the Logging tab, then click enable logging button.
3. On the next screen, select the Kinesis Firehose Delivery Stream that you created already. Finish by clicking create.
4. At this point you should be all set. Wait a few minutes, then start searching in Splunk to make sure your data shows up and that you see field extractions.
Additional Splunk Configuration (optional)
- Map rule ID’s to human readable names (Search Head). Note: You can adapt this for any other ID’s you want to map, too.