How to Build Continuous Integration Into Your Splunk App Development Process

By Hurricane Labs|Published On: March 6th, 2018|

Introducing: Annoying Problem #6,399

Every year around this time I make little resolutions to myself. I’m a rebel so I make them in February or March instead of January 1st. Last year, my resolution was to learn more about, and use, Git for my projects, which I have (for the most part) successfully accomplished. This year, it has been to learn more about automated testing and continuous integration with some of my projects.

Enter: the annoyance.

Normally, all I need to get started on something new is to be really annoyed at the problem I’m trying to solve. In this case, the problem was getting apps submitted to Splunk that weren’t in some violation of their instructions (the amount of submissions that violate security and/or good coding rules happens, a lot). I wanted to spell out my thinking a little bit in the hope of a.) helping someone else at some of the things they get annoyed at and b.) having something my developers will take it into account when they start chasing me with pitchforks. So, here we go.

Let’s put some automation in the process

We build A LOT of Splunk apps, some of which get punted back to us for a variety of reasons. Some are handled by AppInspect, or at least it tells us about them, and others it does not.

I decided to try to minimize/shorten the process of us building and submitting a good app, one that follows their rules and pretty much the rules of the world in general. I would not call this “fully continuous integration”, or anything like that, because it doesn’t really push anywhere; however, it is definitely some of the automation we have needed.

An overview of the process goes like this:

Coder Commits code to GitLab (our internal git server) — CI/CD gitlab-runner gets the configuration from our configuration file (.gilab-ci.yml) and runs all the tests, in order (set in the stages section), and only runs the next test if the one before it passes — it will then notify the developer of the status (email, Slack, whatever you have configured).

We use GitLab internally so this process works with their internal CI/CD system. I am unsure if it works with public GitHub’s, but that’s next on my list for this project/learning expedition. At any rate, at this point I would like to explain how this all works, because I’ve found a lot of information about this topic doesn’t really do that. Fair warning though, I learned how to code before versioning systems were really popular and so I might be a little cranky toward some of this stuff.

The steps to make this work are as follows:

Grab the .gitlab-ci.yml.prod and rename it to .gilab-ci.yml in the root of your repository.
Grab the .gitattributes files and put it in the root of your repository.
Check out the CI/CD – jobs page in your project or wait for the pass/fail email.
If your app contains no Python simply remove the delint and sectests stages.

That is all there is to make this work, provided you have a gitlab-runner (beyond the scope of this post) that can run Docker. It will JUST work at that point. That’s all well and good, but what is this doing underneath? I’m so glad you asked.

Here’s the breakdown of .gitlab-ci.yml and what each section does:

Copy to Clipboard

image: python:2.7

This tells the CI/CD system what sort of thing we’re testing. Since Splunk is on the cutting edge, we’re targeting python 2.7.

stages:
    - lint
    - sectest
    - splunktest

I found the stages section purely by accident. I wanted to make these tests run in order and not run any further until the first phase was passed. This is how to do that.

delint:
    stage: lint
    script:
    - pip install pylint
    - pylint ${CI_PROJECT_DIR}/bin/*.py
    tags: 
        - docker

The “delint” sections sets up our pylint tests so that our code can look good and “standard” -- lots of devs will hate me for this one but I can take it. Style is ok to worry about too.

sectests:
    stage: sectest
    script:
    - pip install bandit
    - bandit -r ${CI_PROJECT_DIR}/bin
    
    tags: 
        - docker

The “sectests” stage runs a neat tool called Bandit that checks our python code for security issues so they can be fixed before releasing. If your code fails this, you probably shouldn’t release it straight away anyway especially if you work for a security company.

splunktest:
    stage: splunktest
    script:
    - wget --output-document splunk-appinspect.tar.gz http://dev.splunk.com/goto/appinspectdownload
    - pip install splunk-appinspect.tar.gz 
    - mkdir /tmp/${CI_PROJECT_NAME}
    - git archive --format tar HEAD | tar -xC /tmp/${CI_PROJECT_NAME}
    - splunk-appinspect inspect /tmp/${CI_PROJECT_NAME} --mode precert
    tags: 
        - docker

Finally there is the “splunktest” stage that downloads the latest version of Splunk’s appinspect, installs it into docker and runs it against the code.

That’s about it from a “how it works” and “how to install it” point of view.

Here are a few final notes

AppInspect has these things called “manual checks” which, in this iteration of the config, won’t cause your code to fail these tests, so you should look at each of them manually. If your code passes the first two phases though you likely won’t have too many manual checks. You’ll also notice the .gitattributes file in the repository. The reason we use the .gitattributes files is so we can have the git archive command ignore . files, which Splunk’s AppInspect hates. That is all that is going on there, so if you have other . files in your repository, please ignore them there. Finally, you’ll likely noticed the tags: – Docker at the end of each stage, this tells the CI system to use Docker to run the tests on this code.

As the warning on the GitHub Page says: “Forewarning: I am very old and very new to the CI/CD world so if I’ve done something horribly wrong here, please feel free to tell me.” Any questions or other name calling, I’m @billford on Twitter, or you can reach me through the @hurricanelabs handle.

About Hurricane Labs

Hurricane Labs is a dynamic Managed Services Provider that unlocks the potential of Splunk and security for diverse enterprises across the United States. With a dedicated, Splunk-focused team and an emphasis on humanity and collaboration, we provide the skills, resources, and results to help make our customers’ lives easier.

For more information, visit www.hurricanelabs.com and follow us on Twitter @hurricanelabs.