Amazon Web Services (AWS) has a really great security feature, called IAM roles, that can be used with EC2 as instance profiles. When you launch an EC2 instance with an instance profile, the IAM role credentials are available to the instance through the metadata service at http://169.254.169.254. Unlike IAM user credentials, IAM role credentials automatically rotate on a schedule (generally every 15 minutes), so even if the credentials are stolen, they’re only good for a short time period.

A downside of IAM roles is that every single process on the system has access to them. If you subscribe to the school of thought that every single instance (or ideally autoscale group) runs exactly one service, this isn’t such a big deal, but if you’re running services in a multi-tenant way, this becomes a problem.

The problem is that your IAM role needs to have the sum of all IAM permissions necessary for all of your services that run on an instance. A common solution for people using Kubernetes or Mesos, or other schedulers is to simply give the IAM roles all AWS permissions. Another common solution is to switch back to IAM users. Neither solution is ideal.

We do subscribe to the school of thought that each autoscale group should run one service. Of course one service doesn’t mean one process, and not every process on an instance needs the full level of privileges that come with the IAM role. Though we don’t do multi-tenancy for real workloads (staging and production) we do multi-tenancy in development, to better utilize resources.

For development (and multi-tenancy) we’re using Docker. We’re also in the process of switching to Docker in staging and production, so we wanted to tackle two problems:

We wanted development to look as much like AWS as possible. For this to work, we want to use IAM roles everywhere. We wanted to strictly scope IAM roles in staging and production to containers that require those privileges.

Enter metadataproxy

We had an idea to build a web service that proxies calls to the metadata service on http://169.254.169.254 and pass through most of the calls to the real metadata service, but capture calls to the IAM endpoints. By capturing the IAM endpoints we can decide which IAM credentials we’ll hand back.

Of course, if we had this idea, we were sure others did too, and they did. First we looked at ec2metaproxy, but we also needed to mock most of the routes outside of AWS. The same author has another project which does exactly that, called aws-mock-metadata. These are both great projects, but they’re written in different languages and it was taking a lot of effort to modify them to make them work together properly.

The general problem wasn’t very involved and the metadata service has a relatively small number of routes, so we decided to build something from scratch, called metadataproxy.

metadataproxy’s primary function is to capture IAM endpoints, figure out which IAM role should be assumed, then return those credentials. It can run in one of two modes. When the MOCK_API setting is set to true, it’ll mock most metadata endpoints. If it’s set to false, it’ll proxy those endpoints through to the metadata service.

To know which IAM roles should be assumed, the metadataproxy has access to the docker socket. When it gets a request, it looks up the container, based on its request IP, finds that container’s environment variables and uses the value of the IAM_ROLE environment variable as the role to assume. It then uses STS to assume the role, caches the credentials in memory (for further requests) and returns them back to the caller. If the credentials cached in memory are set to expire, the proxy will re-assume the credentials.

The metadataproxy can either be run directly on the host, or inside of a Docker container that’s set to the host-only networking. To capture and forward the traffic to the real metadata service, you can use iptables to forward this traffic from the docker0 interface to wherever your service is running. metadataproxy’s documentation has an example of the needed iptables rule.

Contributing to metadataproxy

To contribute to the project, please visit the Github repository and file issues or make pull requests. We’re excited about the release and gladly welcome contribution. Our Github repository has a set of issues being tracked already which are marked easy/medium/hard, to make it easier to pick up any issues that may be relevant to your skillset. The documentation has installation and configuration information that should make it quick and easy to get a version of metadataproxy working in your infrastructure quickly.

Interested in open source work and having a big impact? Lyft is hiring! Drop me a note on Twitter or at ryan.lane@lyft.com.

N E X T → Finding a needle in a haystack