I recently worked with a client who had some log files in Amazon Web Services (AWS) S3 that they wanted to ingest into Splunk. This seemed like a great opportunity to build an example in our lab and document the process for those of you who might be interested in doing the same thing.
For this example, I am going to start by creating a new S3 bucket and uploading some data. You can skip these steps if you have an S3 bucket already and move directly to the section on configuring an IAM user and permissions.
I’ve created a video walking through the process if you prefer this approach:
Otherwise, follow along in the steps below.
Creating some sample data
To make a log file, use a one-line bash script as follows:
I would expect any logs you might ingest to be more useful than these.
Creating an S3 bucket
In the AWS console, search for S3 in the services menu: