NECCDC 2019: Red Team Review

Over this past weekend (March 14-17) I had the opportunity to be a member of the red team for the Northeast Regional of the Collegiate Cyber Defense Competition (NECCDC), held at Champlain College in Burlington, Vermont. For those of you not familiar with CCDC, this yearly event is a cyber security competition where some of the top information security students face off against a red team of industry professionals. While the red team is attacking, students try to keep their systems secure while also maintaining business operations and completing various administration tasks (injects).

Highlights of the Fun!

Check out our “Red Team Update” video on Youtube: https://youtu.be/jOoa4L4NwQE

The theme for this year was business continuity and disaster recovery. Students needed to migrate services from physical to virtual machines, and move them back throughout the competition. They also had to complete other business tasks, like configuring centralized logging (in Splunk, of course), dealing with a stolen laptop, and keeping the guest WiFi up and available. While this is happening, our job as the red team is to maintain persistence, impact scoring, and provide some degree of psychological stress to the student competitions.

Overall, many of the teams did a pretty effective job at containing some of the red team’s implants and callbacks. Teams that have a strong border control policy (at the firewall) generally do a good job at containing (but not necessarily ever removing) a lot of the threats we provide. To make this more interesting, each team had a red team accessible system that could access any system on the team’s network, from behind their firewall. We had the ability to change the IP of this system (which we did on numerous occasions), but this forced teams to deploy more than just a border control strategy to keep us out.

One of my favorite aspects of the NECCDC red team is how well we work together as a group to maintain and share access and persistence. It’s our responsibility to provide a fair and equal experience to each team, and in order to do so effectively we have to collaborate. We each have our specialties – in my case, Splunk and Linux – and often help each other out if we still have access to a team when some of the more easily detectable threats have been identified and remediated.

Advice for Future Competitors

Discussing the blue team strategies with my fellow red teamers, we came up with a number of points of advice for future competitors. Many of these are pretty relevant even outside of the competition landscape as well.

Password Compromise: The red team gets a packet containing all of the passwords that the student teams get at the start of the event. All of these are compromised, and need to be changed.

Machine Compromise: If something pops up on your screen that’s not the result of any action you took (such as a jumbo mouse pointer or a dancing banana man), that means your machine is compromised. If we can make those things happen, you can also know we can log all of your keystrokes, capture your screen, and turn on your webcam and microphone (and make sounds come out of your speakers, too). This access must be removed, and then any passwords used on that system changed (as they are now compromised). Also, removing the sound driver on your machine so that it stops making noise does nothing to address the root cause of someone else having remote control of your system.

Watch SSH Traffic: If there are active SSH connections to your systems that are not you, that system is compromised. Fix the root cause and then change any passwords and SSH keys that were used.

Firewalls Aren’t Everything: While controlling traffic at the border is a great strategy, it doesn’t solve the problem of your systems being compromised – they still are. We can still have mechanisms that are programmed to activate when connectivity is removed, so simply blocking access at the firewall won’t fix that – and you might seem more damage as a result.

Application Behavior: Finally, be familiar with what your applications do and what traffic they produce during normal operation. Verify packages match what is expected. A great example of this: one team reported a default python script in Splunk as being an indicator of compromise. This could have been validated as legitimate by comparing the script to a known clean copy of the Splunk installation (and this is a great strategy for other situations where the tools you are using might be tampered with or otherwise not able to be trusted).

Incident Reports: My Perspective

Additionally, a few of us on the red team (myself included) worked with the competition organizers (white team) to review all of the incident reports and provide feedback and scoring. This worked out really well, since we were most familiar with the incidents we caused and could objectively look at the reports to determine which ones were accurate and well written. This is something that we’ll continue to do in future years.

This is definitely an area of improvement for the majority of the teams. Many of the reports lacked basic information to even allow us to determine what machine was compromised, let alone why. Unfortunately, reports that lack this information are essentially useless to anyone trying to respond to an incident.

Suggestions for Making Better Incident Reports

Plan Ahead: Plan for having to submit incident reports. Create a template and know what information you’ll need to collect, so that it’s ready to go when you need it.

• In the case of CCDC where printed materials are allowed, have each team member prepared with a checklist they can use to record information when an incident occurs. That way you won’t miss out on basic information.
• Having a team member responsible for writing all of the incident reports isn’t a bad idea either.

Gather Relevant Information: Try to gather this information if at all possible:

• Severity of the attack
• Source of the attack
• Target of the attack
• Method of compromise
• How the attack was detected
• Business impact
• Steps taken to remediate
• Lessons learned – future opportunities for improvement

Evidence Matters: Whenever possible, have supporting information. If you have logs showing that something happened, include those in your report. The same goes for screenshots as well – these details combined with an explanation make your message so much more powerful.

Effective Communication: When providing screenshots or console output, highlight relevant information. A great example of where this could have been done better was a team report including an ARP table, showing a duplicate MAC address (as the result of ARP spoofing). Highlighting or circling the duplicate entries would have made this presentation much more clear and effective.

Be Careful When Making Assumptions: Don’t assume the reader is familiar with the issue. Provide background and references as needed.

Executive Summary: Include an executive summary of the issue. Depending on the nature of the incident, your report will be read by a number of people, both technical and non-technical. Those in business leadership roles will be interested in the high-level details as well as business impact (did we lose money, intellectual property, or customer information) and what steps may have been done to prevent the issue from occurring again. Technical individuals will need much more information on what happened, how it was detected, and what was implemented to resolve and prevent the issue. An excellent incident response includes all of this information (kudos to the RIT team for doing an excellent job with this on one of their incident reports).

Watch Your Language: Use professional language. You never know who will read these reports, and what their role might be. The response to an incident is almost more important than the incident itself – things will happen, how you manage that is how you will stand out.

Looking Forward to Next Year!

If everyone can take some of these ideas and include them in their incident reports next year, I think we’ll see a lot more teams getting more credit for completing these reports in future competitions. Since reporting an incident helps offset some of the point loss from red team activity, this is a valuable strategy for minimizing some of the damage that we can do to a team’s score.

I’d like to thank Hurricane Labs for giving me the opportunity to participate in this event, and am looking forward to the 2020 NECCDC event at the University of Maine.