On July 19th, 2019 Capital One got the red flag that every modern company hopes to avoid - their data had been breached. Over 106 million people affected. 140,000 Social Security numbers. 80,000 bank account numbers. 1,000,000 Social Insurance Numbers. Pretty messy right?
Unfortunately, the 19th wasn't when the breach occurred. It turns out that Paige Thompson, aka Erratic, had done the deed between March 22nd and March 23rd 2019. So almost 4 months earlier. In fact, it took an external tip for Capital One to realize something had happened.
Though the former Amazon employee has been arrested and is facing $250k in fines and 5 years in prison...it's left a lot of residual negativity. Why? Because of many of the companies who've suffered data breaches try to brush off the responsibility of hardening their infrastructures and applications to the increased cyber crime.
ANYHOW. You can read more about the case by just asking Google. We won't go into that anymore. We're here to talk about the TECHNICAL side of things.
So first up - what happened?
Capital One had about 700 different S3 buckets that Paige Thompson copied and downloaded.
Second - was this just another case of a misconfigured S3 Bucket Policy?
Nope, not this time. Instead, she was able to access a server that had a misconfigured firewall and do it from there.
Wait how is this possible?
Well first, let's start with getting into the server. The details in terms of how she accessed one of their servers are very little. All we've been told is that it was through a "misconfigured firewall." So something as simple as a crappy security group setup, a web app firewall configuration (Imperva), or network firewall (iptables, ufw, shorewall, etc).
There's been some speculation that the initial breach was done via Server Side Request Forgery, aka SSRF. Which of course is going to lead to the question:
"Cole, what the heck is SSRF??"
Well, in a simplified nutshell, it's when your your servers are running code dumb enough to execute commands that users send up. For example, you might have a website
breachme.com where you can add something like:
url is specified, then your server running
breachme.com will go and grab the file from that url.
So...what happens if we do something like the following?
Well, if you haven't fool-proofed your server, then it's going to let the request user get the local file of the server.
Though this may seem "harmless" (primarily because its an example) the problem is that it allows users to send commands for the server to execute. And so, though the user may not have access to
breachme.com's internal network...the server running it sure as hell does. And if you're executing commands THROUGH the server...guess what? Now you can ask the server to give you things like all of the contents of an S3 bucket that it has access to.
That being said, all we know from Capital One is that they've accepted that it's their fault and that they've closed the hole. Whether it was through SSRF or some other means, unless they give us more, we won't know. The reason is that, contrary to the classic Hollywood Hacker going through a gauntlet, sometimes companies just leave their doors open.
Stone said that while Capital One missed the firewall vulnerability on its own, the bank moved quickly once it did. That certainly was helped by the fact that the hacker allegedly left key identifying information out in the open, Stone said.
And so there's the first thing to do - know what your firewalls allow
Keep a policy or process in place to ensure that ONLY what should be open is open. If you're using AWS resources like Security Groups or Network ACLs, obviously something as simple as a checklist can go a long way...but since many resources are created through automation (i.e. CloudFormation), one can also automate the auditing of these things. Whether that's a home-baked script that looks through what you're creating for crappy openings or something like working in security review during your CI/CD process...there's plenty of simple options to avoid this.
The "funny" part about this breach is that if Capital One had plugged this hole from the get go...nothing else would've happened. And so, quite frankly, it's always shocking to see something that's really not that difficult become the sole reason for a company data breach. Especially one as large as Capital One.
Now if bypassing the firewall was a simple as SSRF, then it means they needed to give their servers reduced access to what they can do on behalf of users. That, or clean up their darn code and check user inputs for malicious commands. At least they didn't let users specify and execute SQL related commands.
Okay, so the hacker is in the box - what happened next?
Well, once you're in (or can execute commands on) an EC2 Instance...a lot can go wrong. You're pretty much walking on the edge of a knife if you let someone get that far. But how did she get into the S3 buckets? To understand how, let's talk about
So, to access AWS services, one way to do so is to be a User. Okay, pretty obvious. But what about if you want to give other things in AWS, for example your Application Servers, the permission to do things like access your S3 buckets? Well, that's what
IAM Roles are for. An
IAM Role has two components:
a) The Trust Policy - what services, or what people, can use this role?
b) The Permissions Policy - what does this role allow?
And so, for example, if you wanted to create an
IAM Role that allowed EC2 instances to access an S3 bucket: first, the role would have a Trust Policy specifying that EC2 (the whole service), or specific instances, can "assume the role." By assume the role, that means that they can use the role's permissions to do things. Second, the permissions policy would allow the service/person/resource that has "assumed the role" to do things on S3, whether that's access one specific buckets...or over 700 in Capital One's case.
The thing is though, once you're on an EC2 Instance with an
IAM Role, you can do a few things to get the credentials:
1. You can query the instance's metadata at http://169.254.169.254/latest/meta-data
Among other things, the
IAM Role with any of its access keys can be found through this. Of course...its only possible if you're IN the instance. OR...if you you can do something like
breachme.com?url=http://169.254.169.254/latest/meta-data and get the data back. Which would of course would be the case if the EC2 instance was running code vulnerable to SSRF.
2. Use the AWS CLI ...
Again, if you've made it on the instance, then if the AWS Command Line Interface is installed, it gets loaded up with credentials from
IAM Roles if they're present. All you have to do is do things THROUGH the instance. Granted, if their Trust Policy was open, Paige may have been able to do it directly.
So the Tl;dr on
IAM Roles are that they're a way to let other resources ACT ON YOUR BEHALF on OTHER RESOURCES.
Now that you understand IAM Roles, we can talk about what Paige Thompson did:
She accessed the Server (EC2 Instance) through an opening in the server firewalls.
Whether it was a security group / ACL, their own custom web application firewalls, SSRF, whatever it was, it was apparently pretty easy to close down as noted in the official complaints and records.
Once in the Server, or had the ability to execute commands on it, she was able to act "as if" she was the server itself.
Since the Server's
IAM Roleallowed for S3 access to those 700+ Buckets, she was able to access them.
From that point forward all she had to do was run the "List Buckets" command and then the "Sync" command from the AWS CLI...
Capital One indicated that this breach will cause nearly $100 to $150 MILLION in costs. Preventing this type of pain is why companies are investing so much into Cloud Infrastructure, DevOps, and Security experts. And just how valuable and cost effective is moving to the cloud? So much that even in the face of more and more cyber security issues, the total public cloud market grew 42% in Q1 of 2019!
Moral of the story: audit your security; keep auditing your security regularly; respect the principle of least privilege for security policies.
(You can checkout the full legal report here)
J Cole Morrisonhttp://start.jcolemorrison.com
Developer Advocate @HashiCorp, DevOps Enthusiast, Startup Lover, Teaching at awsdevops.io