Photo by James Sutton on Unsplash

A couple of months ago we started AWS Elastic Container Registry (ECR) as an alternative to a vendor solution that was giving us a bit of grief.

In an Ops perspective, this was great because ECR becomes virtually maintenance free once you set it up to expire and delete historical images through lifecycle policies. As a developer however, we found the experience to be a bit counterintuitive:

Doing a Docker log-in is hard to understand for those not to familiar with AWS, Cross-Account access and Docker itself. Additionally, ECR access credentials expiring every day makes the initial setup a bit more convoluted.

The Registry URLs is not very friendly ( [ACCOUNTID].dkr.ecr.ap-southeast-2.amazonaws.com ) and there is no way to customise it yet.

) and there is no way to customise it yet. ECR doesn’t currently support policies to configure anonymous access for private network ip ranges.

It was clear we had to do something to make ECR more accessible to the regular developer. Last week we finally stood up a Kubernetes Nginx service that fronts ECR with a custom domain name and allows unauthenticated docker pulls from within VPCs and OpenVPN.

The components of this service are the following:

A ConfigMap with a tokenised Nginx configuration file for processing with fresh ECR credentials and domain name on a schedule.

An Nginx service that proxies the ECR registry.

A CronJob that processes the ConfigMap with new pull secrets on a schedule and bounces the reverse proxy service.

The ConfigMap

The Nginx configuration template (aws-registry-proxy-tpl) is extremely simple. It proxies the ECR registry, forces the host header and sets Docker basic authentication credentials for the request. (Read more here https://docs.docker.com/registry/recipes/nginx/)

The Nginx service

Nothing too special about the proxy service either. It is just a plain Nginx Pod (we happen to use Nginx Plus) that loads the previously processed template (aws-registry-proxy-config) though a ConfigMap volume. The Kubernetes Pod is fronted by a service and an TLS ingress route.

Note the spec includes an imagePullSecrets directive pointing to an aws-registry Kubernetes secret used to authenticate against ECR. This secret gets refreshed with a CronJob as we will see shortly.

The CronJob

Finally we reach the the Kubernetes CronJob that glues all the pieces together:

It calls the aws ecr get-login command to get a fresh ECR token.

command to get a fresh ECR token. It saves the ECR token as aws-registry pull secret so that we can pull our Nginx-Plus container (not needed it you are using a public image).

pull secret so that we can pull our Nginx-Plus container (not needed it you are using a public image). It renders the aws-registry-proxy-tpl ConfigMap template and saves it as the aws-registry-proxy-config ConfigMap used by the Nginx container.

ConfigMap template and saves it as the ConfigMap used by the Nginx container. It creates a minor update to the Nginx Pod deployment so that Kubernetes performs a rolling update on it.

I have been using a Kubernetes Job to trigger the first execution of the template since it seems there is no way to currently invoke a Kubernetes CronJob run on demand.

The Extras

The following is the xynova/aws-kubectl container Dockerfile used to run the CronJob. It basically combines the aws-cli, the kubectl and the gomplate binaries in a container.

For the container to request the docker login commands with the aws-cli, it must run on a Kubernetes node with the following Role attached policy.

Wrapping up

As you can see, it only takes a little bit of routing trickery to bypass some of AWS ECRs limitations. For us this is a big win because now developers can just connect to OpenVPN and directly pull images from registry.shrd.ourdomain.net without having to do any black magic to authenticate against it.

Cheers