Amazon Inspector Review: Great concept, lackluster implementation
Amazon Web Services (AWS) publicly released a new security vulnerability assessment tool called Amazon Inspector. Seeing as how we at Hurricane Labs are heavy users of both AWS and assorted vulnerability assessment tools, it seems like something worth inspecting (sorry). We’re going to take a look at what Inspector is, what’s interesting about it, testing just how ‘special’ it really is, and where it may be headed down the road.
For those of you that don’t have time to read my in-depth review, here is an executive summary: Amazon Inspector is a low-impact, low-cost vulnerability scanner that is beneficial when used in addition to comprehensive vulnerability scanners (such as Nessus or Qualys). Amazon Inspector has limited detection capabilities and a lackluster user interface, however, and should not be used as a substitute to a traditional vulnerability scanning solution.
So, what is Amazon Inspector?
At the most basic level, Amazon Inspector is an agent-based vulnerability assessment tool. What this means is that unlike traditional vulnerability assessment tools, which have centralized servers that scan multiple remote hosts, all scanning is conducted locally and requires the AWS agent to be installed on each target system. Installation is dead simple, and consists of running one executable file.
Like all agent based scanners, Inspector is only supported on a limited set of operating systems. See the official documentation for a full list, but at the time of writing, this includes only the major Linux-based releases like RedHat, CentOS, Ubuntu 14.04, and Amazon Linux. On the Windows side, Windows Server 2008 R2 and 2012 are also supported. This does cover a decent amount of what is commonly used in production, especially in AWS. However, operating system support is a non-issue for traditional vulnerability scanning solutions.
What’s special about this tool?
Amazon claims it “enables you to analyze the behavior of your AWS resources and helps you to identify potential security issues.” It also gathers information about “network, file system, and process activity”, which it sends back to Amazon for analysis and reporting. These are some bold claims, especially inclusion of the word “behavior” which, at least in my opinion, implies a higher level of analysis than just basic checks.
Time to test these claims, here we go
Since Inspector seems to be targeted primarily at DevOps teams, I decided to design a basic Ubuntu 14.04 box with some operating system misconfigurations and a vulnerable WordPress installation. This is so that both halves of DevOps would be tested. The vulnerabilities I targeted are listed at the end of this blog, if you’re looking for specific details.
Next, I had to configure Inspector. This is a fairly simple guided process, consisting of a bit of permissions configuration, creating an Assessment Target, an Assessment Template, and then running an assessment with those previously created objects. First and foremost, Inspector will guide you through setting up an IAM role which gives Inspector limited access to your AWS account.
Then, you’ll create an “Assessment Target”. An Assessment Target is just a list of AWS instances, which is created based on tags that have been assigned to those instances — for example, name. For my testing, I just set this to the test instance I created. An Assessment Template is like a Nessus policy and determines which “Rules” Amazon will use to analyze the hosts, as well as the assessment duration. I selected all of the available rule sets for my test and the recommended duration of 1 hour. Lastly, in order to actually start the scan, just select the template you created, and click the “Run” button.
At this point, Inspector will begin gathering data from the agents on whatever targets you specified. You can expand the Assessment Run you created, and click Status, which will show you how many agents are active and how much telemetry has been obtained from those agents. As far as system resource usage goes, Inspector is extremely minimal. You won’t see any of the CPU/Network usage spikes like you would with Nessus or Qualys. Inspector makes very little noise on both the system and the network — this may be one of its best features.
After the assessment is finished, the results will automatically populate in the “Notable Findings” section of the Inspector Dashboard. If you’re used to navigating any of the AWS dashboards, the Inspector interface will feel very familiar. The interface is simple, but in my opinion it leaves quite a bit to be desired. My biggest complaint is that there is no way to see descriptions of all of the vulnerabilities returned by a particular scan in one place. It is possible to see a list of 10 findings at once, which contain CVE ID numbers. But, unless you’re in the practice of memorizing CVE numbers, that information is of very little value to you.
This is what the Dashboard looks like, if you’re curious.
It is possible to export a list of the findings as a CSV, but that’s not very human readable and it still doesn’t include a description of the CVEs. The other rule sets which check for system configuration information have more readable descriptions, like “Instance i-4a5bb0d0 is configured to allow users to log in with root credentials over SSH.” This is a bit better, but again it would be nice if it would use the tagged name instead of the instance ID.
Another seemingly obvious feature that is missing from Inspector is the ability to schedule scans. Since you’ll have to go elsewhere for actual descriptions of the CVEs that Inspector has identified, it seems as though Amazon Inspector is only valuable if you can pipe the data it generates into other tools, whether it be something custom written or a tool like Splunk.
While viewing results is an important part of a vulnerability assessment tool, actually finding results in the first place is even more important. In general, Amazon inspector will likely find less vulnerabilities than the traditional tools. In this case, Inspector was able to find a lot of vulnerabilities in the specific versions of tools that were installed on the local system, but missed some serious vulnerabilities and misconfigurations, like the outdated WordPress 3.5 installation.
Shown here is the full comparison of the scanner results.
The tool still has a few kinks
As you can see, overall, Inspector found less vulnerabilities than any other scan tool I tested. It completely missed the outdated version of WordPress I had installed, which every single other scanner I used managed to find. In fact, it didn’t find ANY web vulnerabilities whatsoever. In my opinion, this is a huge oversight on a product that is marketed primarily towards DevOps teams. While operating system patches are an important part of vulnerability management, vendor and custom installed software is equally important, or even more important depending on whether those services are publicly available.
In order to get some additional visibility into what the aws-agent was actually doing during the test, I installed sysdig on the test machine I built and had it write to a file during the assessment run. Based on that, it seems that the aws-agent reads through pretty much everything in /var/lib/dpkg/info/ in order to get an accurate picture of what software is installed locally. Why don’t they just take a quick look in /var/www/ and check for common CMS installs? The agent also looks for configuration files in /etc/ to check for things like AllowRootLogin over SSH or passwordless sudo. I know this tool is relatively new, but I can’t understand why they would leave out actually checking for custom or vendor applications.
What I really like about Inspector
Now that I’ve identified most of my complaints about Inspector, I can talk about the things about it that I really like. The first thing that I really like is cost. If you have any experience with AWS, you will know that their pricing models can get a bit complex, but an Amazon Inspector scan will never cost you more than 30 cents. For that price, you could run thousands of scans every month and it would still be far cheaper than Nessus or Qualys. For how little it costs, there’s almost no reason you wouldn’t have it running regularly on any host that’s being used in development or in production.
Speaking of running on machines that are actually in production, Inspector is great for that. While running a Nessus or Qualys scan on your production web servers is almost definitely a bad idea, running Inspector on your production machines even during peak hours is actually a good idea. It won’t cause a ton of extra CPU or Network usage, and you’ll have better data for Inspector to use.
There’s a lot of potential here
In conclusion, Amazon Inspector is a tool with a lot of potential. As of now, I cannot recommend the exclusive use of Inspector as a vulnerability scanning tool, but it is definitely worth installing on at least a few of your AWS resources, as it is cheap and easy to use. In the future, I imagine that Amazon will develop more advanced rule sets and be able to deliver on their promise of analyzing the behavior of AWS resources. It definitely doesn’t do this now, but knowing Amazon, it probably will eventually.
Thank you for reading!
Details of the vulnerabilities targeted to test Amazon’s “claims”.
About Hurricane Labs
Hurricane Labs is a dynamic Managed Services Provider that unlocks the potential of Splunk and security for diverse enterprises across the United States. With a dedicated, Splunk-focused team and an emphasis on humanity and collaboration, we provide the skills, resources, and results to help make our customers’ lives easier.
For more information, visit www.hurricanelabs.com and follow us on Twitter @hurricanelabs.
