The following subsections explain the individual components of this system.

Authentication with IAM Identities

The first thing to note is that EKS uses Identity and Access Management (IAM) identities for authentication. An IAM identity can be an IAM user or an IAM role.

In practice this means that the bearer token, which clients include in the request, must contain an IAM identity (in the form of an ARN). The authentication service on the server-side then checks whether this IAM identity corresponds to a valid user of the cluster.

Because the bearer token includes an IAM identity, I call it IAM Identity Token in the above figure.

The idea of the EKS team behind using IAM identities for authentication is to not have to define a new set of users and credentials for the Kubernetes cluster, but to reuse existing IAM identities.

AWS IAM Authenticator

As mentioned, the authentication decision in EKS is made by a webhook service that gets called by the API server. This webhook service is implemented by a tool called AWS IAM Authenticator.

The AWS IAM Authenticator is an open-source project maintained by a Kubernetes Special Interest Group (SIG) and is maintained on GitHub.

Note that the same AWS IAM Authenticator tool is also used in the client-side part of the EKS authentication mechanism, which will be discussed here. In fact, you come in touch with the AWS IAM Authenticator only on the client-side, because on the server-side EKS does it for you.

The AWS IAM Authenticator tool provides the executable aws-iam-authenticator which provides several sub-commands. One of these sub-commands is the following:

aws-iam-authenticator server

This command starts up the EKS webhook authentication service. This means that when you create your EKS cluster, EKS runs this command for you (probably on a master node), and configures the API server to use the provided endpoint as the target for the webhook token authentication method.

Now, whenever the API server receives a request, it forwards the IAM identity token in the request to the webhook service. The webhook service first verifies whether the contained IAM identity is a valid IAM identity with the AWS IAM service.

When this is done, the webhook service consults an object called aws-auth ConfigMap to check whether the IAM identity corresponds to a valid user of the cluster. The aws-auth ConfigMap is crucial to the server-side authentication mechanism, and is the place where you have to define the IAM identities that you want to grant access to the cluster. All this will be the subject of the next section.

aws-auth ConfigMap

The aws-auth ConfigMap contains a mapping of IAM identities to Kubernetes users. It is the place where you have to define who and what can access your EKS cluster. How to do this will be explained later, because first of all, we should know what a ConfigMap is at all.

What is a ConfigMap?

A ConfigMap is a Kubernetes API object. This means it is an object that can exist in a cluster like a Pod or a Deployment. The specification of the ConfigMap object can be found here.

A ConfigMap has a very simple internal structure. It simply consists of a set of key/value pairs. All keys are strings and the values can be either strings or binary data.

The purpose of a ConfigMap is to contain configuration information. It has a similar role like configuration files in local programs. ConfigMaps are created for and used by pods and system components.

The YAML definition of a simple ConfigMap looks like this:

kind: ConfigMap

apiVersion: v1

metadata:

name: my-configmap

data:

key1: value1

key2: value2

key3: value3

As you can see, this ConfigMap consists of three key/value pairs, all of which are strings. Key/value pairs are contained in the data field (there exists also the binaryData field that can contain key/value pairs with binary data values).

As any other Kubernetes object, you can create this ConfigMap in the cluster with kubectl , for example:

kubectl apply -f configmap.yml

The ConfigMap exists now in the cluster and can be accessed by cluster components (for example, pods) by the name that you gave it (in this case my-configmap ).

You can view all the existing ConfigMaps in your cluster as as follows:

kubectl get configmaps --all-namespaces

The aws-auth ConfigMap

The aws-auth ConfigMap is a ConfigMap called aws-auth that must exist in every EKS cluster (specifically, in the kube-system namespace). As mentioned, it is read by the AWS IAM Authenticator webhook service to read the list of IAM identities that should be granted access to the cluster.

EKS does a lot of work for you, but it doesn’t create the aws-auth ConfigMap for you! You are responsible for creating and maintaining this object

However, if you created an EKS cluster following the EKS Getting Started guide, then you already created this ConfigMap! Specifically, in the “launch worker nodes” step, you were requested to insert the IAM role ARN of your worker nodes into a ConfigMap template, and then apply this template to the cluster.

The ConfigMap template that you had to modify looks like this:

apiVersion: v1

kind: ConfigMap

metadata:

name: aws-auth

namespace: kube-system

data:

mapRoles: |

- rolearn: <WorkerNodesRoleARN>

username: system:node:{{EC2PrivateDNSName}}

groups:

- system:bootstrappers

- system:nodes

Let’s look at it more closely.

Format

This ConfigMap contains a single key/value pair in the data field. Don’t be confused by the pipe character ( | ) and the subsequent lines that look like they are part of the YAML document. The | is a YAML multiline string indicator, which means that everything that follows is to be interpreted as a single multiline string.

Thus, the key is called mapRoles and the value is the following string:

- rolearn: <WorkerNodesRoleARN>

username: system:node:{{EC2PrivateDNSName}}

groups:

- system:bootstrappers

- system:nodes

Both of these strings (key and value) do have no meaning to Kubernetes. For Kubernetes, they are just a key string and a value string. However, both key and value, do have a meaning to the AWS IAM Authenticator.

In fact, it is the format that the AWS IAM Authenticator requires the aws-auth ConfigMap data to be in. This configuration format is defined here in the AWS IAM Authenticator documentation.

Content

So, what does this key/value pair mean to the AWS IAM Authenticator? Here it is again:

mapRoles: |

- rolearn: <WorkerNodesRoleARN>

username: system:node:{{EC2PrivateDNSName}}

groups:

- system:bootstrappers

- system:nodes

In the AWS IAM Authenticator configuration format specification we can read that each entry of mapRoles maps an IAM role to a username and a set of groups.

Consequently, there must be an entry for each IAM role that should be granted access to the cluster. The mapped username and groups define a Kubernetes “subject”. This subject has no meaning to the authentication system, but it will be the basis for the subsequent RBAC authorisation step. This will be explained in a later section.

In the above example, the worker nodes IAM role is mapped to “user” named system:node:<DNSName> that is in the groups system:bootstrappers and system:nodes (the config format specification also defines that {{EC2PrivateDNSName}} will be resolved to the DNS name of the instance where the request comes from). This username and groups define which actions the RBAC authoriser will allow this request to perform.

The specification also defines a key called mapUsers that is used to specify mappings for IAM users (rather than of IAM roles). An extended aws-auth ConfigMap with additional IAM users and IAM roles will be presented below.

Why did we have to do this?

Why did we have to create this ConfigMap? The EKS Getting Started guide says that it enables our worker nodes to join the cluster. But why?

Another way to formulate this is that it enables the worker nodes to “talk” to the API server in the first place.

Following the EKS Getting Started guide, we created the worker nodes as an EC2 autoscaling group. At this point, they were not yet associated with the cluster. In order to be added as worker nodes to the cluster, they must request this from the API server (via kubelet which is already running on them). As mentioned, every request to the API server must include an IAM identity as a bearer token. The IAM identity that the worker nodes include in their request is the IAM role that we assigned to the worker nodes during the creation process. This role is called NodeInstanceRole in the official CloudFormation template which is used in the EKS Getting Started guide.

However, up to now, this IAM role was not listed in the aws-auth ConfigMap, so the AWS IAM Authenticator did not authenticate these requests from the worker nodes (in fact, up to this point, the aws-auth ConfigMap didn’t yet even exist).

By adding the IAM role of the worker nodes to the aws-auth ConfigMap, we fix this and we finally make the requests from the worker nodes being authenticated by the AWS IAM Authenticator. This allows the worker nodes to finally talk to the API server and be added to the cluster.

An extended aws-auth ConfigMap

By listing the IAM worker nodes role in the aws-auth ConfigMap, we essentially defined the first “user” that is allowed to talk to the API server. If we want to add additional cluster users, then we have to add them as new entries to the aws-auth ConfigMap.

In the following example, I edited the aws-auth ConfigMap to include an additional IAM role and IAM user (highlighted in bold):

apiVersion: v1

kind: ConfigMap

metadata:

name: aws-auth

namespace: kube-system

data:

mapRoles: |

- rolearn: <WorkerNodesRoleARN>

username: system:node:{{EC2PrivateDNSName}}

groups:

- system:bootstrappers

- system:nodes

- rolearn: <LambdaRoleARN>

username: lambda

groups:

- system:masters

mapUsers: |

- userarn: <UserARN>

username: user

groups:

- system:masters

As you can see, I had to add the IAM role mapping to the already existing value of the mapRoles key, and the IAM user mapping to the separate mapUsers key. All this is defined in AWS IAM Authenticator configuration format specification.

I freely chose a username for both mappings, and I associated them with the system:masters group, which is a pre-defined group that is interpreted by the RBAC authoriser to allow full access to the cluster (more on that below).

How to edit the aws-auth ConfigMap?

Once created, you can easily edit the aws-auth ConfigMap with the following command:

kubectl edit -n kube-system configmap/aws-auth

When you save the edited file, your changes will be automatically applied.

What about the cluster creator?

The IAM identity that created the EKS cluster is automatically “hardwired” in the AWS IAM Authenticator. This means that this IAM identity is recognised and authenticated (and mapped to a user in the system:masters group) by the AWS IAM Authenticator without being listed in the aws-auth ConfigMap. This is the reason that after creating the EKS cluster, you can already access the cluster with kubectl without doing any configurations (if you use the same IAM identity for kubectl ). However, any other IAM identities that you want to grant access to the cluster, you must explicitly configure in the aws-auth ConfigMap.

Authorisation

In the aws-auth ConfigMap we define mappings from IAM identities to “usernames” and “groups”. These usernames and groups define so-called “subjects” which are the basis for the RBAC authorisation step that is carried out for each request after successful authentication.

In general, authorisation systems are driven by policies that define who can do what. For Kubernetes RBAC authorisation, these policies are specified by a set of Kubernetes API objects called Role and RoleBinding (there is also ClusterRole and ClusterRoleBinding, which have cluster scope rather than namespace scope). These objects have to be specifically created for the RBAC authoriser, and they are only used by the RBAC authoriser.

In short, a role consists of one or more rules. Each rule defines a set of allowed Kubernetes actions (for example, get pods ). A role binding binds a role to a one or more subjects. And a subject can be a username or a group.

So, if in the aws-auth ConfigMap, you define a specific IAM identity to be mapped to a username user in the group group , then the RBAC authoriser will allow this request if the requested Kubernetes action is listed in some role which is bound (through a role binding) to either the username user or the group group .

For example, in the last section we defined a mapping from an IAM user to a user in the system:masters group. The system:masters group is a default group. There is also an RBAC default role called cluster-admin , which allows all possible Kubernetes actions. Furthermore, there exists a default role binding that binds the cluster-admin role to the system:masters group. Thus when the RBAC authoriser handles a request from this user, it applies the cluster-admin role, because this role is bound to the system:masters group.

This works exactly the same with non-default usernames and groups (that is, usernames and group names that you choose yourself), but in this case you have to define and create your own custom Role and RoleBinding (ClusterRole and ClusterRoleBinding) objects. As soon as you create these objects, they will be considered by the RBAC authoriser for all subsequent requests.

You can create these objects like any other Kubernetes API objects by defining them according to their specification in a YAML file and then applying them with kubectl apply . Guides about creating roles and role bindings can be found here and here, respectively

In general, you can choose any string for usernames and groups, but the prefix system: is reserved for Kubernetes system use.

You can view all the existing roles and role bindings in your cluster with the following commands:

kubectl get roles --all-namespaces

kubectl get clusterroles

kubectl get rolebindings --all-namespaces

kubectl get clusterrolebindings

You can view the content of a specific role or role binding like this:

kubectl describe clusterrole cluster-admin

kubectl describe clusterrolebinding cluster-admin

As you can see, authentication and authorisation are independent systems, but nevertheless they work in tandem. The output of the IAM authenticator is the input to the RBAC authoriser. You can define any usernames and groups in the authenticator, but you have to apply meaning to them in the authoriser.

This concludes our discussion of the server-side of the EKS authentication mechanism. Let’s now go over to the client-side part.

EKS Authentication: Client-Side

As we have seen, the EKS authentication mechanism requires that all requests to the API server contain an IAM identity in a bearer token. In the server-side part we have discussed how this token is verified on the server-side. Now, in the client-side part we are going to discuss how the clients obtain an IAM identity and generate such a token.

The overall process is summarised in the following figure. Note that is process applies to all types of clients, be it your local machine, and EC2 instance, a Lambda function, or a worker node within the cluster:

EKS client authentication (client-side).

The following subsections will explain all the main components of this process.

Kubernetes Client Library

At the centre of the above diagram is a Kubernetes client library. Kubernetes client libraries wrap HTTP requests for the Kubernetes API calls into functions that can be called from code. This allows programatic access to Kubernetes (that is, you can write a program that can do everything that you can do with kubectl ).

In fact, tools like kubectl and kubelet are nothing else than programs that access Kubernetes through a client library (and kubectl providing you a nice command-line interface through which you can invoke the individual API requests). So we can say, that every request that is ever made to the API server is made through a Kubernetes client library (except if you access the API server with raw HTTP calls, but we can neglect this for our purposes).

There are a number of officially supported Kubernetes client libraries in different programming languages. They are all hosted in the following GitHub organisation:

There is one more very important Kubernetes client library, namely the official Kubernetes Go client library. It is hosted in the following GitHub repository:

The Go client library is important because it is the one used by official Kubernetes tools like kubectl and kubelet (but you can also use it for your own programs). If there are new Kubernetes feature, this library is usually the first to implement them).

kubeconfig File

Where do all the Kubernetes client libraries get the cluster information from?, For example, the URL of the API server? The answer is from a kubeconfig file. One of the first things that you usually do when using a Kubernetes client library is to read a kubeconfig file.

A kubeconfig file is a YAML configuration file for Kubernetes cluster access. The default kubeconfig file is ~/.kube/config , but a kubeconfig file can have any name, and you can have multiple of them.

Note that after creating an EKS cluster, you can automatically create or update the kubeconfig file with the following command: aws eks update-kubeconfig --name <ClusterName>

The syntax format for kubeconfig files is described here in the Kubernetes documentation. In short, a kubeconfig file contains three main pieces of information:

Clusters

Contexts

Users

The clusters section contains two mandatory pieces of information: (1) the API server URL, and (2) the API server certificate authority (CA) certificate. The CA certificate is used by the client to verify the server certificate, that is, to verify the identity, of the API server (this is server authentication, which is the opposite to client authentication, the topic of this article).

The contexts section defines triples of clusters, namespaces, and users for easy reference.

Finally, the users section defines credentials for authenticating to the API server. For an EKS cluster, the users section must be in a specific format, which is described in the next section.

Credential Plugins Feature

First of all, let’s see how the kubeconfig file users section looks like for an EKS cluster:

users:

- name: <ClusterARN>

user:

exec:

apiVersion: client.authentication.k8s.io/v1alpha1

command: aws-iam-authenticator

args:

- token

- -i

- <ClusterName>

The crucial point is the exec property. This property is provided by a feature of the Go client library called credential plugins, which is described here.

The exec property allows to define an external command that generates and returns credentials. The returned credentials can be in one of two forms: a bearer token or a private key and certificate (this is explained here).

The credential plugins feature must be implemented by the Kubernetes client libraries, because the client libraries read the kubeconfig file and must “understand” the exec property. It is also the client library that executes the external command and then reads the output value of the command.

Note that “credential plugins” is a relatively new feature (Kubernetes v1.10), and currently not all Kubernetes client libraries implement it. The official Go client library implements it, and since kubectl and kubelet use the Go client library, you can also use this feature with these tools. However, for other client libraries, you need to check whether they already support it.

For an EKS cluster, the external exec command is as follows:

aws-iam-authenticator token -i <ClusterName>

Let’s execute this command to see what it does. The output is as follows:

{

"kind": "ExecCredential",

"apiVersion": "client.authentication.k8s.io/v1alpha1",

"status": {

"token": <TOKEN>

}

}

The command prints some JSON. The format of this JSON is specified by the credential plugin feature (see here). The most important part is the token property. This is the bearer token that has to be included in the API server request and is the basis for the authentication against the API server.

The next section explains how the command generates this token.

AWS IAM Authenticator on the Client-Side

As you might have noticed, the exec command above is provided by the same AWS IAM Authenticator tool that was already used on the server-side. Here is the client-side command again:

aws-iam-authenticator token -i <ClusterName>

The difference to the server-side is that the token sub-command is used, rather than the server sub-command.

As mentioned, the produced token must contain an IAM identity. However, we cannot directly see it in the output, because the token is encoded to match the HTTP format requirements. Fortunately, AWS IAM Authenticator also provides another sub-command for decoding a token:

aws-iam-authenticator verify -i <ClusterName> -t <TOKEN>

This command prints a Go struct (because AWS IAM Authenticator is written in Go) that contains the ARN of the IAM identity encoded in the token.

Note that the -i value can be any string, but must be the same for the token and verify sub-commands (explained here), and that tokens expire after 15 minutes ( verify sub-command will fail after this time).

The big question now is: where does this command get the IAM identity from?

Which IAM identity?

The answer is, the first IAM identity that is encountered in the AWS credential chain that is configured on the host system.The AWS credential chain is explained in detail here. In short, it is a sequence of locations that are checked by AWS tools and SDKs for an IAM identity. The default credential chain is as follows:

AWS_ACCESS_KEY_ID , AWS_SECRET_ACCESS_KEY , and AWS_PROFILE environment variables ~/.aws/credentials file Assigned IAM role, if the host is an EC2 instance or Lambda function

The aws-iam-authenticator token command uses this same credential chain to determine the IAM identity to include in the token. You can always check which IAM identity this is, by executing the following AWS CLI command:

aws sts get-caller-identity

This already concludes our tour of the client-side part of the EKS authentication mechanism. As mentioned, after the aws-iam-authenticator token command executed, the client library reads the generated token and includes it as a bearer token in the request to the API server.

Client-Side Authentication Ingredients

I mentioned that the client-side part of the authentication mechanism is always the same, no matter from where the requests are made. Let’s dig into this by considering some example scenarios.

The following list summarises the components that need to be present in the environment of a client for being able to make requests to the API server:

Kubernetes client library or tool (e.g. kubectl or kubelet )

or ) kubeconfig file

aws-iam-authenticator executable

executable IAM identity

Let’s go through some scenarios on different example hosts: a local machine, an EC2 instance, a Lambda function, and an EKS worker node in the cluster.

Local Machine

On your local machine, you have most likely kubectl installed. To access an EKS cluster, you have to update your kubeconfig file, which you can do with the following command:

aws eks update-kubeconfig --name <ClusterName>

You also need to install aws-iam-auhenticator , which you can do with the following command (see here):

go get -u github.com/kubernetes-sigs/aws-iam-authenticator/cmd/aws-iam-authenticator

Finally, there must be an IAM identity in the credential chain, which is likely your main IAM user in the ~/.aws/credentials file.

EC2 Instance

If you want to access an EKS cluster from an EC2 instance, it’s likely that you want programmatic access to the cluster (that is, deploy a program that accesses the cluster). In this case you don’t need kubectl , but a Kubernetes client library. You typically include this library in your program by specifying it as a dependency of your main program.

You need to create a kubeconfig file and install the AWS IAM Authenticator executable on the EC2 instance, which you can do with the same commands shown above.

If you assigned an IAM role for your EC2 instance, then this role will be used by the aws-iam-authenticator token command. If the EC2 instance does not have an IAM role, then you have to specify an IAM identity, either as an environment variable or by creating an ~/.aws/credentials file on the EC2 instance.

Lambda Function

In this scenario, you create a Lambda function that accesses your EKS cluster. Since the access to Kubernetes is programmatic, you don’t need kubectl too, but a client library, which you specify as a dependency of your Lambda function code.

The Lambda function needs to have a kubeconfig file and the aws-iam-authenticator executable in its filesystem at run time. You can add these files to the artefact folder that will be zipped and uploaded to S3 by the aws cloudformation package command (if you use CloudFormation). From this location, the files will then be deployed to the Lambda function’s runtime environment each time the function is invoked.

Note that since you have to provide aws-iam-authenticator as a binary to the Lambda function, it needs to be compiled for the target platform used by your Lambda function. Fortunately, AWS IAM Authenticator is written in Go, and you can easily cross-compile it. To compile for a Linux target platform, as used by Lambda, you can set the variables GOARCH=amd64 and GOOS=linux and then compile the source code with go build .

Each Lambda function needs to have an IAM role, and, by default, aws-iam-authenticator will use this role for generating the authentication token.

EKS Worker Node

For a worker node in an EKS cluster (created with the official CloudFormation template) all the requirements are already there (that’s actually the reason why they are able to join the cluster). Since this is an interesting topic, let’s inspect a worker node in more detail in the next section.

Inspecting a Worker Node

Log in to a worker node of an existing EKS cluster with SSH (note that for this to work, you might have to add a inbound rule allowing SSH traffic to the security group of the worker node):

ssh -i my-key.pem ec2-user@XXXX.compute.amazonaws.com

Once you are logged in, you can check that all the client-side authentication ingredients are there:

Verify that kubelet is installed:

kubelet --version

See the kubeconfig file used by kubelet :

cat /var/lib/kubelet/kubeconfig

Verify that aws-iam-authenticator is installed:

aws-iam-authenticator --help

See the IAM identity that is used by aws-iam-authenticator :

aws sts get-caller-identity

As you can see, everything is there and looks similar as on your local machine. The point is to show that the client-side part of EKS authentication really works the same for any type of client, no matter whether it’s your local machine, an EC2 instance, a Lambda function, or a worker node within the cluster.

I hope that reading this article gave you a good understanding of how client authentication works on EKS. If you have any questions about EKS authentication, please let me know in the comments.