Even though the above was made public this week, the Platform Team at Mettle always wants to be ahead of the curve. Therefore, the remainder of this blog post will talk about our journey to upgrade our clusters to v3.

Preparation

The following sections detail the prep work required before starting to upgrade.

HelmRelease: Default all to v2

As preparation, we made sure that all our HelmReleases specified the helm version they use, this was done by adding the following:

spec:

chart:

name: backend

repository: https://example.storage.googleapis.com

releaseName: account-balance

helmVersion: v2

Helm Operator: Allow both v2 and v3 charts

We need to set the following options on our helm operator instance so that it’s ready to support both Helm v2 and v3:

- --enabled-helm-versions=v2,v3

Helm: Install both v2 and v3 locally

We are going to need both helm 2 and 3 for this so its best to have both installed, you can do this by executing the following:

brew install helm@2

brew install helm

cd /usr/local/bin

ln -s /usr/local/opt/helm@2/bin/tiller tiller

ln -s /usr/local/opt/helm@2/bin/helm helm2

ln -s helm helm3

From now on you need to use helm3 for Helm v3 and helm2 for Helm v2.

Upgrade Workflow

The remainder of this blog post describes the workflow taken to upgrade.

Stop Helm Operator

We need to stop the Helm Operator in the cluster to allow for the existing helm releases inside the cluster to be upgraded. We did this by scaling the replica count to zero within the deployment.

Upgrading helm releases in the cluster

To upgrade the helm releases already existing in the cluster we needed to use https://github.com/helm/helm-2to3.

Firstly we need to list all the existing helm releases inside the cluster using:

Remove Tiller

Once we have upgraded the existing releases in the cluster to v3, there is no need for Tiller to be running anymore. We did this by scaling the replica count to zero within the deployment.

Upgrade HelmReleases to v3

We then updated all our HelmReleases specifying v3 as the helm version, this was done by adding the following:



chart:

name: backend

repository:

releaseName: account-balance

helmVersion: v3 spec:chart:name: backendrepository: https://example.storage.googleapis.com releaseName: account-balancehelmVersion: v3

Merge these changes into your cluster and wait for all HelmRelease manifests in the cluster to be using v3 before starting the Helm Operator.

You can use kubectl get hr -A -o yaml | grep ‘helmVersion: v2’ to make sure all the v2 references have been fully removed.

Start Helm Operator

The final step is to scale back up the helm operator deployment to 1 replica and only allow v3 releases, this was done by adding the following:

- --enabled-helm-versions=v3

Only allowing v3 here is key as the cluster no longer has Tiller so v2 is not a valid option anymore!

Merge the Helm Operator change

It is recommended to merge the Helm Operator PR first and make sure the pod(s) are successfully running.

Merge the HelmRelease changes

Now merge the HelmRelease changes to helmVersion:v3 and now its time to validate that the migration is working

Sanity check

Once turning back on the Helm Operator it starts to upgrade the HelmReleases. When describing a HelmRelease you will see such output as:

Running upgrade for Helm release 'platinum-docs' in 'web'.

You want to wait until all HelmRelease resources say:

Release was successful for Helm release ...

You can validate this by running:

watch "kubectl get hr -A | grep -v 'Release was successful for Helm release'"

You are expecting the above command to eventually be an empty list, once you see this the migration has been successful.

Summary

Overall this was a pretty painless process and took about an hour to perform the upgrade from start to finish (per environment). I would like to thank Stefan Prodan (https://twitter.com/stefanprodan) once again for being the sounding board on my proposed upgrade strategy.