Apply AWS KMS encrypted Secrets as it is, in Kubernetes h3poteto Follow Mar 23 · 4 min read

I have developed a Custom Controller for Kubernetes, which name is KMS Secrets. This controller decrypts AWSK KMS encrypted data and generate Secret resources, so you can apply KMS encrypted data definitions as it is.

Why?

Sometimes I don’t want to commit Secret resources to Git as is, because Secret has raw (base64 encoded) strings. This request will often occur in GitOps workflow.

I often use AWS, and I have created Kubernetes clusters in AWS. So, I want to encrypt secret data using AWS KMS, and commit encrypted data to Git. But in normal operation, I have to decrypt these Secrets before apply it; Therefore, I think that it is very useful if Kubernetes allows encrypted Secrets.

How to use it

Please define a KMSSecret resource:

apiVersion: secret.h3poteto.dev/v1beta1

kind: KMSSecret

metadata:

name: mysecret

namespace: mynamespace

spec:

encryptedData:

# Encrypted string using AWS KMS

API_KEY: AQICAHh2iCEGE2e6vdC+w6dQ4hRIyahEPE...

PASSWORD: AQICAHh2iCEGE2e6vdC+w6dQ4hRIyahEPE...

# AWS Region where the key is located

region: us-east-1

template:

metadata:

labels:

"h3poteto.dev/custom-labels": my-label

annotations:

"h3poteto.dev/annotations": my-annotation

In this time, please write encrypted string in spec.encryptedData . You can get encrypted string using awscli :

$ aws kms encrypt --key-id 1asdf3-rsdf... --plaintext "apikey" --query CiphertextBlob --output text

AQICAHh2iCEGE2e6vdC+w6dQ4hRIyahEPE...

In generally, you have to provide base64 encoded string in Secrets. But, now please encode plain text, don’t provide base64 encoded string. This encrypted string is automatically encoded base64 when decrypt.

After you apply this KMSSecret resource, a Secret resource is generated:

apiVersion: v1

data:

API_KEY: YXBpa2V5

PASSWORD: cGFzc3dvcmQ=

kind: Secret

metadata:

annotations:

h3poteto.dev/annotations: my-annotation

creationTimestamp: "2020-03-18T07:27:06Z"

labels:

h3poteto.dev/custom-labels: my-label

name: mysecret

namespace: mynamespace

ownerReferences:

- apiVersion: secret.h3poteto.dev/v1beta1

blockOwnerDeletion: true

controller: true

kind: KMSSecret

name: mysecret

uid: deac9220-68e9-11ea-8182-0658b210029a

resourceVersion: "10673189"

selfLink: /api/v1/namespaces/mynamespace/secrets/mysecret

uid: dec32aea-68e9-11ea-ae9a-0a561536d7cc

type: Opaque

How to install

I prepared helm chart, so please helm install.

$ helm repo add h3poteto-stable https://h3poteto.github.io/charts/stable

$ helm install h3poteto-stable/kms-secrets --name kms-secrets

Permission for AWS

KMS Secrets will access AWS KMS to decrypt, so the permission is required. Please give a IAM role to this controller with required permissions.

Probably you are using IAM Role for Service Account, if you have created Kubernetes clusters in AWS. If it, give eks.amazonaws.com/role-arn to annotations in ServiceAccount.

You can specify this annotations when helm install:

$ helm install h3poteto-stable/kms-secrets --name kms-secrets --set rbac.serviceAccount.annotations=your-iam-role-name

At this time, attachme this policy to your-iam-role-name :

{

"Version": "2012-10-17",

"Statement": [

{

"Sid": "Allow use of the key",

"Effect": "Allow",

"Action": [

"kms:Decrypt",

"kms:DescribeKey"

],

"Resource": "*"

}

]

}

Related OSS

Sealed Secrets

KMS Secret is similar to SealedSecrets. SealedSecrets is very famous in this use case.

This OSS provide CLI and custom controller. We can encrypt data using CLI, and apply it, then custom controller decrypts it and generate Secrets.

It is very good, but SealedSecrets generates a key to encrypt/decrypt in controller. So we have to manage this key, and backup it.

I don’t want manage keys, and I don’t want store it, so I’m using AWS KMS.

kubesec

This OSS provide CLI, and it can encrypt/decrypt Secrets using AWS KMS and Google Cloud KMS. But we have to decrypt it before apply.

Kubernetes External Secrets

This OSS take another approach. It take secret data from AWS Secrets Manager, so user have to store secret data in AWS Secrets Manager. And after user apply ExternalSecret resource, the custom controller get decrypted data from Secrets Manager, and generate Secrets.

In Conclusion

I released the first version of KMS Secrets. It is one of resolution how to manage Secret resources in GitOps.

There are many tools to encrypt Secrets using AWS KMS, for example, kubesec, yaml_vault, and awscli. I think it is good that create KMSSecret resource using yaml_vault and apply it in Kubernetes cluster which is installed KMS Secrets.

Now this controller supports only AWS KMS, but I consider to support Google Cloud KMS. If you have a request, please create an issue on GitHub.