How did the teams respond?
The good news is that the majority of the teams took note of the compromise of their AWS environments. Additionally, they worked together to try to remediate any issues.
First, it was necessary for teams to identify how we got in and how we were accessing their accounts. Fortunately, the competition infrastructure team was nice enough to provide each team with functional AWS CloudTrail logging. This gave teams visibility into all of the Red Team-related account shenanigans. By watching these logs for suspicious activities, they could identify accounts that were used maliciously. Subsequently, they could change credentials and revoke access keys. Teams that were most effective at removing our access entirely typically employed a strategy of keeping a continuous watch of the CloudTrail logs.
By the end of the event, around half of the teams had completely removed all of the (known) Red Team access into their AWS accounts. Of the three teams that placed 1st, 2nd, and 3rd, all of them regained control of their AWS account by the morning of the second day. Kudos to these teams for quickly figuring out a way to navigate the attacks and secure their environments.
AWS Security Recommendations from this NECCDC 2023 Experience:
Let’s put ourselves into the student teams’ shoes. You’ve just been given control of a pre-existing AWS environment, you have no idea of the state of security, and need to secure it. What do you do?
- Change Default Credentials: It’s a good idea to assume passwords are compromised. Additionally, assume that you don’t know if any credentials are outside of the control of trusted and authorized individuals. Best practice is to force a password reset for all users. You can communicate new credentials to users via a mechanism where you can validate whether or not they’re ending up in the right hands.
- Audit User Accounts: Access to the AWS management console should only go to users who need it. Does your sales team need access to the AWS console? Probably not. The principle of least privilege is key here.
- Audit IAM Roles: Review all of the IAM roles present in your environment, especially custom ones. Roles that allow wide permissions changes should be tightly controlled. Also, limited permissions should be assigned to users based on their job function.
- Maintain Control of Your AWS Root Account – And Don’t Ever Use It: Normal administration should be accomplished by using accounts with assigned permissions. The root account should be stored securely for emergency use only, and not day-to-day operations.
- Audit AWS Access Keys: Review all AWS access keys that have been created. Revoke any that aren’t actively being used, and rotate any that are known to be used legitimately. Consider alternative API access mechanisms that don’t grant the same level of administrative control. Establish policies for access key rotation.
- Audit S3 Buckets: S3 buckets should generally not be publicly accessible. Many breaches and information disclosure attacks have involved S3 buckets that were unintentionally made publicly available.
- Enable CloudTrail: Ensure you are capturing AWS CloudTrail events. Even better – forward those events to a SIEM such as Splunk and set up alerting for unauthorized access!
- Assume Your Systems are Compromised: Don’t trust anything. If the AWS environment itself has been compromised, there’s a good chance that there are also outstanding issues with the servers and software running on your cloud infrastructure as well.
Due to the complexity of AWS and the number of services, this list is not comprehensive. However, it is a good starting point to locking down access to your cloud environment.
How does this blog post – and NECCDC 2023 – add value to your life?
As a Splunk consultant and implementer at Hurricane Labs, it’s easy to see why someone may ask “why do you do this stuff?” The honest answer is because I enjoy a good challenge and a learning experience. Red-teaming exercises, in particular, are where the fun begins. It’s exhilarating to discover new ways to gain access and maintain control of multiple systems while also expanding knowledge. Recently, I took on the task of infiltrating 10 identical corporate networks hosted entirely on AWS, all while attempting to escalate privileges and keep my access under wraps. Also, I can confidently say that the student teams and I gained a deeper understanding of AWS and its various security controls along the way.
Special thanks to the incredible support from Hurricane Labs. They graciously covered my travel and accommodation costs for the recent NECCDC weekend, and have empowered me to participate in and support these types of events. I’ve had the honor of participating in regional events since 2018. Looking forward to next year’s event already!