NECCDC 2023: Red Team Adventures
As cybersecurity enthusiasts, we don’t miss any chance to participate in events that challenge our skills and support up-and-coming security professionals in the process. That’s why it was a thrill to represent Hurricane Labs at the Northeast regional event for the Collegiate Cyber Defense Competition (NECCDC 2023). Meredith Kasper and I were a part of the Red Team. The purpose of this team is to load up the competition with targeted attacks. It’s the perfect opportunity for student competitors to get the full experience of triaging attacks against their infrastructure.
This year’s event was especially notable because all of the infrastructure was hosted on Amazon Web Services (AWS). The scope also included the AWS management console, which immediately piqued my interest as an attractive target.
In this blog post, I’ll share my expert approach for red-teaming an AWS environment. As we dive into the intricacies of AWS security, I’ll also equip you with the best security tips to safeguard your own AWS environment.
Mounting an AWS Attack
Our first access to the competition infrastructure is typically achieved via a credential compromise – similar to many of the attacks out there today. During NECCDC, students are typically given a list of default credentials at the start of the event. As red team members, we often find a way to get a hold of these too.
The educational opportunity here is pretty simple: assume default credentials are compromised.
The most interesting column in this sheet for me was the one labeled “AWS”.
First, I had to determine the AWS account IDs. This task was accomplished with a simple curl request to the AWS metadata URL on the instances we had access to. From there, I began logging into every account in the sheet and got a sense of what I could do from every role.
- Every user had the ability to log into the AWS management console
- Users on the sales team had full access to a public S3 bucket
- Users in the engineer role had full access to EC2 permissions
- Users in the security role had a custom permission scoped to managing IAM roles, but this didn’t allow adding new roles to existing or new users
- One user (scornelissen) had additional permissions beyond the engineering role
While this base level of access was interesting, many of the first attempts to escalate permissions didn’t work. Specifically, I wasn’t able to add higher permissions to an existing user or group, or create a new user and assign any role. While this was a setback, I went searching for other options that would allow me to gain more access.
AWS Access Keys
The discovery that the users on the security team had significant permissions around IAM roles was interesting. I ended up discovering that it was possible to generate an AWS access key for the currently logged in user. However, I could also generate an AWS access key for any other user in the organization. This meant that, while I couldn’t add new users, I could hijack any existing account and assume the permissions of that user. I could also change the password for the account as well.
With these access keys, it was then possible to use the AWS API to modify existing accounts. I could also reset passwords for users with higher levels of access.
Here’s a screen recording of the entire privilege escalation process. This shows an account takeover of a team’s administrative account from a user with lower permissions.
This was the breakthrough I needed to take control of every team’s AWS account, and further establish persistence into the AWS environment. We had a stack of AWS access keys, and went to work.
We worked together as Red Team members to inform teams of our presence, such as making some “updates” to their public facing website:
Obviously, having your public-facing website look like this isn’t a great image to show to potential new clients.
Since the goal of the Red Team is to differentiate teams, we need to impact scoring. We accomplished this by making some “improvements to AWS security groups”…
…as well as saving the teams on their AWS costs by downsizing instances and then shutting them off (because we like saving money on the Red Team).
How did the teams respond?
The good news is that the majority of the teams took note of the compromise of their AWS environments. Additionally, they worked together to try to remediate any issues.
First, it was necessary for teams to identify how we got in and how we were accessing their accounts. Fortunately, the competition infrastructure team was nice enough to provide each team with functional AWS CloudTrail logging. This gave teams visibility into all of the Red Team-related account shenanigans. By watching these logs for suspicious activities, they could identify accounts that were used maliciously. Subsequently, they could change credentials and revoke access keys. Teams that were most effective at removing our access entirely typically employed a strategy of keeping a continuous watch of the CloudTrail logs.
By the end of the event, around half of the teams had completely removed all of the (known) Red Team access into their AWS accounts. Of the three teams that placed 1st, 2nd, and 3rd, all of them regained control of their AWS account by the morning of the second day. Kudos to these teams for quickly figuring out a way to navigate the attacks and secure their environments.
AWS Security Recommendations from this NECCDC 2023 Experience:
Let’s put ourselves into the student teams’ shoes. You’ve just been given control of a pre-existing AWS environment, you have no idea of the state of security, and need to secure it. What do you do?
- Change Default Credentials: It’s a good idea to assume passwords are compromised. Additionally, assume that you don’t know if any credentials are outside of the control of trusted and authorized individuals. Best practice is to force a password reset for all users. You can communicate new credentials to users via a mechanism where you can validate whether or not they’re ending up in the right hands.
- Audit User Accounts: Access to the AWS management console should only go to users who need it. Does your sales team need access to the AWS console? Probably not. The principle of least privilege is key here.
- Audit IAM Roles: Review all of the IAM roles present in your environment, especially custom ones. Roles that allow wide permissions changes should be tightly controlled. Also, limited permissions should be assigned to users based on their job function.
- Maintain Control of Your AWS Root Account – And Don’t Ever Use It: Normal administration should be accomplished by using accounts with assigned permissions. The root account should be stored securely for emergency use only, and not day-to-day operations.
- Audit AWS Access Keys: Review all AWS access keys that have been created. Revoke any that aren’t actively being used, and rotate any that are known to be used legitimately. Consider alternative API access mechanisms that don’t grant the same level of administrative control. Establish policies for access key rotation.
- Audit S3 Buckets: S3 buckets should generally not be publicly accessible. Many breaches and information disclosure attacks have involved S3 buckets that were unintentionally made publicly available.
- Enable CloudTrail: Ensure you are capturing AWS CloudTrail events. Even better – forward those events to a SIEM such as Splunk and set up alerting for unauthorized access!
- Assume Your Systems are Compromised: Don’t trust anything. If the AWS environment itself has been compromised, there’s a good chance that there are also outstanding issues with the servers and software running on your cloud infrastructure as well.
Due to the complexity of AWS and the number of services, this list is not comprehensive. However, it is a good starting point to locking down access to your cloud environment.
How does this blog post – and NECCDC 2023 – add value to your life?
As a Splunk consultant and implementer at Hurricane Labs, it’s easy to see why someone may ask “why do you do this stuff?” The honest answer is because I enjoy a good challenge and a learning experience. Red-teaming exercises, in particular, are where the fun begins. It’s exhilarating to discover new ways to gain access and maintain control of multiple systems while also expanding knowledge. Recently, I took on the task of infiltrating 10 identical corporate networks hosted entirely on AWS, all while attempting to escalate privileges and keep my access under wraps. Also, I can confidently say that the student teams and I gained a deeper understanding of AWS and its various security controls along the way.
Special thanks to the incredible support from Hurricane Labs. They graciously covered my travel and accommodation costs for the recent NECCDC weekend, and have empowered me to participate in and support these types of events. I’ve had the honor of participating in regional events since 2018. Looking forward to next year’s event already!
About Hurricane Labs
Hurricane Labs is a dynamic Managed Services Provider that unlocks the potential of Splunk and security for diverse enterprises across the United States. With a dedicated, Splunk-focused team and an emphasis on humanity and collaboration, we provide the skills, resources, and results to help make our customers’ lives easier.
For more information, visit www.hurricanelabs.com and follow us on Twitter @hurricanelabs.