Inside Splunk Certification Exam Development

Have you ever wondered what goes into the development of the Splunk Certification exams? If so, you’re in the right place. 

In this post, I will discuss the Splunk exam development process, how Splunk Trust folks got involved in writing exam questions, and what to keep in mind when you’re the one taking an exam. 

How did the Splunk Trust members get involved?

Earlier this year, Splunk announced the Ideas Portal, where Splunk Community members can propose suggestions for improving the product. These ideas will get voted on by other Splunk Users, and votes will help determine which ones get selected. One such idea was to have SplunkTrust members review certification exams to help ensure the quality and technical accuracy of exam questions and content. When Jason Hupka reached out to the SplunkTrust for volunteers to assist with the effort, I jumped on the opportunity immediately.

I’ve since been involved with the initial development of the Splunk Cloud Certified Admin exam (currently in beta), as well as a refresh for the Splunk Core Certified Consultant exam.

What does the exam development process look like?

Before going through the exam development process, I had no idea what happened behind the scenes. There are actually quite a few steps that are involved from initial conception to exam development. At a high level, the process looks something like this: 

1. The exam blueprint/outline is developed.
2. A team of subject matter experts is assembled to develop the exam content. This consists of Splunk employees familiar with the exam topics, the instructors who develop and teach the associated course content, and external third-party subject matter experts (e.g., SplunkTrust members). 
3. The exam blueprint is broken down into writing assignments for each question writer. Each exam writer will generally work on 15-20 questions for an exam.
4. Throughout the writing process, the exam development team meets regularly to review all of the questions that have been written by team members. Each question and answer choice is reviewed, discussed/debated, and revised as appropriate. 
5. A second group of subject matter experts reviews the questions developed by the question writing team, making additional revisions as needed.
6. Questions are reviewed by technical writers to ensure appropriate formatting and adherence to Splunk’s style guidelines. 
7. A final review of the questions is performed. 
8. The exam enters a beta testing phase, where eligible candidates take the exam. Scores are not immediately available at this time.
9. Once a large enough sample size has taken the beta exam, the performance of each question is reviewed. Questions that perform poorly are removed from the final exam form. It is at this point that the passing score is determined, and beta exam participants are notified of their pass/fail status. 
10. The exam is moved from beta to a final release form; candidates now receive scores immediately. 
11. Exams are reviewed and updated several times a year. New questions are written, and they enter a beta testing process where they are included on exams but not counted in scoring until a similar review process takes place. 

As you can see, there’s quite a bit more to exam development than someone simply writing a question and putting it on the exam. My involvement in the process at this point has focused around steps 2-4 and 11 of the above outline, and I’ll be working with the certification team on step 9 for the Splunk Cloud Certified Admin exam in 2021. 

How are questions developed?

Currently, Splunk Certification exams are directly tied to the coursework associated with any certification exam. In other words, in order for a question to exist on an exam, it must be explicitly covered in the Splunk Education course materials, such as the course slides, the lecture material from the course instructor, or the lab exercises. As exam writers, we won’t be writing questions based on our professional experience if that material does not also exist in the coursework. This ensures that the exam material reflects the knowledge that a candidate would be expected to have after completing the courses and labs associated with a certification track. 

Splunk provides several comprehensive resources for understanding the topics that will be covered on each exam. These include the Certification Exam Study Guide and Candidate Handbook that go into the details about what you should know when preparing for an exam. Be sure to review the test blueprints for any exam you’re looking to take, as those list the topics that will be covered by the exam. You’ll generally find these to be very closely supported by the course materials for your Splunk Education classes. 

Writing good questions isn’t easy!

When writing an exam question, we need to keep several things in mind. These include, but are not limited to, the following concepts:

• All questions need to be multiple choice and have four possible answers.
• Only one answer for each multiple choice question must be correct (obviously). 
• The other answer choices may be plausible, but cannot be correct.
• Questions need to be clear and concise. 
• When possible, questions should be positive in nature (“which of the following is true” versus “all of these except”). Sometimes it is unavoidable. 
• When possible, keep answer choices homogeneous in length and structure.
• Understand that not everyone is a native English speaker. Keep wording simple and clear, avoiding words with multiple meanings. Avoid colloquialisms or idioms that may be unfamiliar to a non-native English speaker. 

Since questions are written by individuals, it’s easy to overlook some of these guidelines when developing content. That’s where peer review comes in.

Item review sessions

Throughout the question development process, the exam development team meets to go through each and every question with several objectives in mind:

• Ensure that the question is clear and unambiguous.
• Ensure that there is only one correct answer to the question. 
• Ensure that there are not edge-cases that may allow someone to argue that an intended incorrect answer is actually correct.
• Ensure that all word choices are appropriate for an international exam audience. 

If you’ve taken a multiple-choice exam, I’m sure you’ve internally debated the merits of individual answer choices when trying to pick the correct answer (I do this all the time myself). When writing questions, we’re not only checking the right answer and moving on, but also checking every single answer choice. Wrong answers are often included for very specific reasons (to test knowledge of the topic for the question), but every wrong answer choice will have something to ensure that it is definitely incorrect. 

As you can imagine, when a team of 6-12 subject matter experts evaluate and debate these questions, there can be quite a bit of discussion. As questions are revised to make them more clear, the entire meaning of a question can change–which can also result in other answer choices becoming correct. Before accepting a question and passing it to the next stage of review, multiple subject matter experts must agree that the question is appropriate before we proceed. For some questions, the process is quick and easy, but it’s not uncommon for us to spend quite some time revising a question before we’re confident in its content and presentation. 

The mind of a question author

Let’s walk through the development of an example exam question. I’ll note that this isn’t an actual exam question for any Splunk exam, but it will still give you a good idea of the rationale behind the item writing process.

As an exam writer, suppose I’ve been asked to write a question for the Splunk Certified Consultant exam that covers preparing for an onsite engagement. I’ll first reference the course material and review what expectations are covered in the training. For this example, let’s assume there’s a slide discussing how to dress appropriately for conducting onsite work at a client in the course material. 

First, I must decide what I want to test with the question. For some questions, this could be a simple topic, where knowing a fact will allow you to answer the question. However, I tend to prefer questions that assess whether or not a candidate is able to take the information learned in the course and apply it to a new situation. 

For this example, I want to set up a scenario and ask the exam candidate to put themselves in the shoes of the character in the exam. My first draft of the question might look something like this: 

Brian is a Splunk consultant who is traveling to his first onsite engagement with his client, a financial services company located in a major east-coast United States city. Which of these items would be best for making a positive first impression with his client?

A – Open-toe sandals

B – Pants with a belt

C – Cargo shorts

D – A Splunk T-Shirt

Let’s break down the question. First, what do we know?

• The consultant is new to Professional Services work (which is likely what we would expect for the target exam candidate). 
• The Professional Services work is being done onsite (this story apparently takes place before or after the COVID pandemic). 
• The client is in the financial services industry.
• The client is located on the east coast of the United States (we can assume somewhere like New York City, for example).
• The consultant wants to make a positive first impression.

At this point, the exam candidate would be thinking back to their training and what material was covered in the coursework. In my fictional class example, I’m expecting there to be some material about how appropriate attire varies based on the industry and area where a client is located, and that doing some research on the client ahead of time is a good idea. The expectation here is that a financial services client in New York is likely a much more formal work environment than a new tech startup in San Francisco. 

Now, let’s look at the possible answers, and discuss the merits of each:

• A – Open-toe sandals: this doesn’t seem like it would be appropriate in any professional work environment.
• B – Pants with a belt: wearing pants is generally recommended, so it seems plausible.
• C – Cargo shorts: while it may be acceptable for some environments, it’s probably not the best choice for this engagement.
• D – A Splunk T-Shirt: we know you have a bunch of Splunk T-Shirts and you wear them every day. This seems plausible, too.

With this in mind, you’ve narrowed down your choices to B (pants) and D (Splunk T-shirt). Now, you have to pick the best option. My intent when writing the above question was for B to be the correct answer with D being a strong distractor, but there’s a potential argument that a Splunk T-Shirt (especially with a jacket, as seen in the linked video) may be perfectly acceptable for these types of clients. 

This is likely something that would come up during the question review process, and we’d look to revise that answer choice to avoid that argument and make the intended answer (B) the only reasonable choice. The final exam question would likely be similar to, but not identical to, the form presented here. 

Remember, the answer isn’t always C

When taking an exam (outside of the beta period for a new exam), you won’t always see the same questions as every other candidate. When reviewing questions, we flag those that may provide too much information or ask the same question as another question on the exam. Answer choices are also randomized when the exam is delivered, so know the content.

We’re humans and not perfect

When taking an exam, you may run across questions that don’t seem quite right. Fortunately, Splunk has a Certification Exam Challenge Form available to allow you to submit feedback on exam content. 

If you need to go this route, try to provide as much information as you can remember about the issue you observed, which will help the certification team work with you to understand the issue and work to make the exam better. They have the ability to review the exact exam you took as well as your interaction with the exam, so any information you can provide is helpful for them to understand your concerns. 

A great learning experience

While it’s one thing to take an exam, it’s a completely different experience to be part of the writing process. Even as an experienced Splunk administrator, I’ve learned quite a bit about the product while developing exam content, working with other reviewers, and confirming the accuracy of exam questions. 

If you’re qualified and want to check out our work, I’d encourage you to sit for the beta period of the Splunk Cloud Certified Admin exam, which should be available through Spring 2021. 

Overall, this has been a positive experience for me, and I’m hoping to continue helping the certification team in the exam development and review process in the future.