TL;DR

Kubernetes supports EBS Persistent Volumes by default. It also supports EFS Persistent Volumes by the external efs-provisioner.

This article introduces EFS Persistent Volumes (EFS PVs) and explains how we can use it and migrate to it.

EBS vs EFS

The article “When to Choose Amazon EFS” says:

Amazon EFS provides shared file storage for use with compute instances in the AWS Cloud and on-premises servers. Applications that require shared file access can use Amazon EFS for reliable file storage delivering high aggregate throughput to thousands of clients simultaneously. Amazon EBS is a cloud block storage service that provides direct access from a single Amazon EC2 instance to a dedicated storage volume. Applications that require persistent dedicated block access for a single host can use Amazon EBS as a highly available, low-latency block storage solution.

Pros

EBS PV provides only ReadWriteOnce access mode. EFS PV provides ReadWriteMany access mode as well.

An EFS file system can be accessed from multiple availability zones and it is valuable for a multi-AZ cluster.

You pay only for the used amount. When it is difficult to estimate the amount used by an application, you may choose EFS.

For example, you requested a volume of 100 GB but the application used only 10 GB data, cost of EBS and EFS are followings:

EBS 100 GB costs $10/month. ($0.1/GB/month)

EFS 10 GB costs $3/month. ($0.3/GB/month)

Cons

An EFS file system is a NFS file system. Some applications such as databases require a block device and may not work with an EFS file system.

EFS does not support snapshot. You need to setup a rsync job for backup.

Build the stack

1. Create an EFS file system

The following resources are required for EFS PVs.

An EFS file system

EFS mount targets in each AZ

A security group of the EFS mount targets

You can create the resources on AWS Management Console.

EFS in AWS Management Console

If you are using kops and Terraform, you can refer the kops managed resources in Terraform using the tag as like:

data "aws_vpc" "kops_vpc" {

tags = "${map("kubernetes.io/cluster/hello.k8s.local", "owned")}"

} data "aws_subnet_ids" "kops_subnets" {

vpc_id = "${data.aws_vpc.kops_vpc.id}"

} data "aws_security_group" "kops_nodes" {

tags {

Name = "nodes.hello.k8s.local"

}

} resource "aws_efs_file_system" "efs_provisioner" {

tags {

Name = "efs.hello.k8s.local"

}

} resource "aws_efs_mount_target" "efs_provisioner" {

count = "${length(data.aws_subnet_ids.kops_subnets.ids)}"

file_system_id = "${aws_efs_file_system.efs_provisioner.id}"

subnet_id = "${data.aws_subnet_ids.kops_subnets.ids[count.index]}"

# This example uses the nodes security group

# but you should create a dedicated one for better security.

security_groups = ["${aws_security_group.k8s_nodes.id}"]

} output "efs_provisoner_fsid" {

value = "${aws_efs_file_system.efs_provisioner.id}"

}

See also full configuration at int128/kops-terraform-starter.

Then apply the configuration.

terraform apply

2. Deploy the efs-provisioner

You can deploy the efs-provisioner from the Helm chart.

Here we use Helmfile and create helmfile.yaml as follows:

releases:

- name: efs-provisioner

namespace: kube-system

chart: stable/efs-provisioner

values:

- efsProvisioner:

efsFileSystemId: {{ requiredEnv "efs_provisoner_fsid" }}

awsRegion: {{ requiredEnv "AWS_DEFAULT_REGION" }}

path: /

storageClass:

name: efs

Then deploy it.

export AWS_DEFAULT_REGION=us-west-2

export efs_provisoner_fsid="$(terraform output efs_provisoner_fsid)"

helmfile sync

You can switch the default Storage Class from gp2 to efs as follows:

kubectl patch storageclass efs -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "true"}}}' kubectl patch storageclass gp2 -p '{"metadata": {"annotations": {"storageclass.beta.kubernetes.io/is-default-class": null}}}'

3. Deploy an application

Here we deploy Jenkins from the Helm chart for example. It accepts a Storage Class by Persistence.StorageClass parameter.

Deploy Jenkins by the following helmfile.yaml :

releases:

- name: jenkins

namespace: devops

chart: stable/jenkins

values:

- Master:

ServiceType: ClusterIP

HostName: jenkins.dev.example.com

Persistence:

StorageClass: efs

and then an EFS PV should be created as like:

Kubernetes Dashboard

Note that the EFS PV will be deleted when you run helm delete jenkins .

To avoid deletion, you need to set the Reclaim Policy to Retain as follows:

kubectl patch pv pvc-1e3fcdea-8c2b-11e8-b795-0669efa944a4 -p '{"spec": {"persistentVolumeReclaimPolicy": "Retain"}}'

Migrate data from EBS to EFS

If you already have an application with EBS PV(s), you can migrate it to EFS PV(s). This section shows migration steps.

1. Create a new PVC

Create a new PVC with the efs Storage Class.

apiVersion: v1

kind: PersistentVolumeClaim

metadata:

name: new-pvc

spec:

accessModes:

- ReadWriteOnce

resources:

requests:

storage: 8Gi

storageClassName: efs

And then a PV and EFS mount will be created. As well as you can deploy the application with a new PVC by Helm.

Now there are the following resources on the Kubernetes cluster.

New app → New PVC → EFS PV → EFS mount

Old app → Old PVC → EBS PV → EBS volume

2. Copy files

Stop the applications by setting replicas to 0.

kubectl scale --replicas=0 deployment/new-app kubectl scale --replicas=0 deployment/old-app

Make sure no pod of the application is running.

Create the migration Job.

apiVersion: batch/v1

kind: Job

metadata:

name: migration

spec:

template:

metadata:

name: migration

spec:

containers:

- name: migration

image: alpine

command: ["/bin/sh"]

args:

- -c

- apk add rsync && rsync -av --delete /old/ /new/ && ls -la /new

volumeMounts:

- name: old-volume

mountPath: /old

readOnly: true

- name: new-volume

mountPath: /new

volumes:

- name: old-volume

persistentVolumeClaim:

claimName: old-pvc

- name: new-volume

persistentVolumeClaim:

claimName: new-pvc

restartPolicy: Never

backoffLimit: 0

It may take a long time while copying all files. Finally the EBS PV and EFS PV have same files.

3. Start the application with new PVC

Start the application by setting replicas to 1 or more.

kubectl scale --replicas=1 deployment/new-app

The application should be up with the migrated data.

4. Delete the application with old PVC

Delete the resources of old application.

kubectl delete deployment/old-app pvc/old-pvc

If the application is managed by Helm, just delete the release.

helm delete --purge old-app

Now there is the following resource on the Kubernetes cluster.

New app → New PVC → EFS PV → EFS mount

Conclusion

You can use EFS PVs by setting up an EFS file system and the efs-provisioner. As well as you can migrate from EBS PVs to EFS PVs by creating a migration job.