An OpenVPN with RDS MySql backend deployment that can withstand an AWS AZ crash.

I am sure many of you out there already use OpenVPN to securely access to your VPCs. I am also certain that just like myself, you have tried to figure out a way to deploy the supported AWS Marketplace AMIs in a highly available or fault tolerant way.

I must say it is unfortunate that AWS doesn’t provide Live migration for the EC2 service when it is in fact offered by other cloud providers such as Azure and Google Cloud Platform. In a nutshell, this feature would allow machines to freely fail in a Rack and be automatically transferred along with their state to a different Rack or Datacenter all together (best-effort).

Although I agree that such a feature would introduce potential non-deterministic characteristics to some workloads, we still have plenty of Pet Server Applications like OpenVPN that are tightly bound to some type of network or storage state.

UPDATE: As Luke Youngblood mentions in the comments below, AWS provides a CloudWatch event you can potentially listen to and recover the EC2 attachments.

A typical workaround in AWS is to code application startup as a Launch Configuration and run servers on an multi-AZ Auto Scale Group of 1 instance. To make this technique work for the Marketplace OpenVPN AMI, we needed to ensure the setup met the following requirements:

Being able to offload OpenVPN’s configuration database to a multi-AZ RDS instance. Being able to automatically update OpenVPN’s DNS record to resolve to a brand new Elastic IP on first boot.

Luckily, this was all achievable through AWS instance UserData and IAM permissions. The following scripts describe our current implementation with a couple of highlights:

We create a unique Route53 hosted zone for OpenVPN that just contains its own record and we give the OpenVPN instance role the ability to update records in that Zone. Although this can sound unnecessary, it is the only way to guarantee that the OpenVPN server role cant be used maliciously and take down all other records in a shared zone.

that just contains its own record and we give the OpenVPN instance role the ability to update records in that Zone. Although this can sound unnecessary, it is the only way to guarantee that the OpenVPN server role cant be used maliciously and take down all other records in a shared zone. We pass-in an offline encrypted secret to the instance containing the database password. We do this to prevent the database password being stored in clear text in the terraform state file.

Launch Configuration and Auto Scale Group

The following UserData Terraform template migrates the OpenVPN local database into RDS, updates its DNS record and sets up some default useful runtime settings.

Launch Configuration UserData script.

Finally, the IAM policy makes the automation possible.

IAM policy for OpenVPN server

Wrapping up

It’s not rocket science, but it took a little bit of fiddling around with OpenVPN to get these scripts working. I hope this serves you as a reference for whenever you are in need of setting up a new OpenVPN server in AWS.

Cheers