Ingesting a CSV file into Splunk
CSV files are a common method of data exchange. A comma-separated values (CSV) file is a text file that uses commas to separate or delimit values into “fields.” Each line of the file is called a “record” and every record within a file has the same sequence of fields. CSV files may have a header line at the beginning to describe each field, but it’s not mandatory.
What do CSV files look like?
This is an example of a header line and 4 records from a CSV file export of a Splunk search.
We have 4 fields–they are labeled: host, source, sourcetype, and component. Each of the records has those 4 fields and while the fields are the same, the contents of the fields can vary from record-to-record.
Most spreadsheet applications like Excel can export to the CSV format, and it’s often used as a common format for users of different spreadsheet applications. Splunk can export the results of a search to a CSV file for use in other applications. Lookup files in Splunk can be CSV format as well. Many other applications can import or export data in CSV format.
CSV files can be used to transfer data from another system or application into Splunk, when other methods of data exchange like syslog or an API are not available or are not suitable.
On to ingesting
There are two methods of ingesting:
- Upload with Splunk Web: This is a one-time process done manually by the user. (Note that uploading via Splunk Web has a 500 Mb limit on file size.)
- Monitor from a filesystem with a UF or other forwarder: This method is for on-going ingestion over a period of time and may not require any manual intervention by the user once setup.
Data Upload with Splunk Web
In the Splunk Web UI, you’ll navigate to: Settings -> Add Data -> Upload
You should then see the “Select Source” page as shown in the screen capture below:
Either drag and drop your CSV file into the box shown in the screenshot above, or click on the “Select File” button to open a dialog box that lets you browse your local computer’s file system and then select a file to upload.
Set the source type and other options.
Select host setting and index.
Review what you’ve set and click Submit.
The data from your CSV file is now searchable in Splunk.
Monitor inputs for CSV files
You will need to create an app with an inputs.conf that specifies the file or path to monitor.
The inputs.conf file will contain something similar to this:
Create an accompanying props.conf file:
Either create the app directly on the system ingesting the file, or create it on the Deployment Server and deploy it to the system ingesting the file, whether that’s Splunk Enterprise or a system with the Splunk Universal Forwarder installed. Once Splunkd is restarted on that system, Splunk will begin to ingest the new file.
CSV files are a quick and easy way to import or export data from Splunk. It’s a very common format and you’ll find that many other applications can use it as well. Choose it the next time someone comes to you with structured data exported from a spreadsheet or other application that needs to be ingested and searchable with Splunk.
About Hurricane Labs
Hurricane Labs is a dynamic Managed Services Provider that unlocks the potential of Splunk and security for diverse enterprises across the United States. With a dedicated, Splunk-focused team and an emphasis on humanity and collaboration, we provide the skills, resources, and results to help make our customers’ lives easier.
For more information, visit www.hurricanelabs.com and follow us on Twitter @hurricanelabs.