UPDATE: For a newer Istio control plane upgrade method using the canary upgrade flow see the Safe and sound canary upgrade for your Istio control plane post.

When we added support for Istio’s service mesh in the Pipeline platform, we experienced first hand how the deployment and management of Istio can become increasingly complex. We realized that we weren’t the only ones managing Istio with Helm dealing with these problems - that demand was emerging for an Istio operator (e.g. #9333). We decided to build an Istio operator of our own, and more than a month ago we open sourced it. Our goal was to simplify the deployment and management of Istio for everyone.

The first version of our operator was capable of installing and managing Istio 1.0.x versions. Then, as soon as Istio 1.1 became available, we updated our operator to support that version as well (we also currently support Istio 1.1.1).

More than simply facing the challenges of Istio deployment and management, many users had additional trouble with upgrades (e.g. #12154, #12658, #12659, #12664, #12755, #12801, #12855 and #12866). With Helm, issues even arose while upgrading between minor versions. However, since the most recent major release of Istio, upgrading between major versions has been the larger challenge. As of now, manual upgrade is the recommended approach.

Since our operator was able to deploy and manage both Istio 1.0.x and 1.1.x versions, we have been working on making upgrades easier and more seamless between these versions. Today, we released a version of our operator which greatly simplifies the control plane upgrade between Istio versions.

Control plane upgrade with Istio operator 🔗︎

Let us suppose that we have a Kubernetes cluster with Istio 1.0.5, and we would like to upgrade our Istio components to Istio version 1.1.1. Here are the steps we need to perform to accomplish this with our operator:

Deploy a version of our operator which supports Istio 1.1.x Apply a Custom Resource using Istio 1.1.1 components

It really is that easy!

What happens is that once the operator discerns that the Custom Resource it’s watching has changed, it reconciles all Istio-related components in order to perform a control plane upgrade.

The upgrade process described above is between major Istio versions. Upgrading between minor Istio versions is even easier. You only need to perform the second of those two steps, i.e. apply a Custom Resource with the desired Istio version set for Istio’s components, and the operator will do the rest for you.





Try it out! 🔗︎

Minikube v0.33.1+ or Kubernetes 1.10.0+

KUBECONFIG set to an existing Kubernetes cluster

I created a Kubernetes cluster on GKE for this demo with our Pipeline platform. If you wish you could also create one on any of the six cloud providers we support or on-premise using Pipeline for free.

If you already have Istio 1.0.x installed on your cluster you can skip the next section and can jump right to Deploy sample BookInfo application.

Install Istio 1.0.5 🔗︎

We install Istio with our operator, so first we need to check out the release-1.0 branch of our operator (this branch supports Istio versions before 1.1.0):

$ git clone git@github.com:banzaicloud/istio-operator.git $ git checkout release-1.0

Install Istio Operator with make

Simply run the following make goal from the project root to install the operator:

$ make deploy

This command will install a Custom Resource Definition in the cluster, and will deploy the operator to the istio-system namespace. As is typical of operators, this will allow you to specify your Istio configurations to a Kubernetes Custom Resource.

Install Istio Operator with Helm

Alternatively, if you just can’t let go of Helm completely, you can deploy the operator using a Helm chart, which is available in the Banzai Cloud stable Helm repo:

$ helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com $ helm install --name = istio-operator --namespace = istio-system --set-string operator.image.tag = 0.0.10 banzaicloud-stable/istio-operator

Note: As of now, the 0.0.10 tag is the latest version of our operator to support Istio versions 1.0.x.

Apply the Custom Resource

Once you’ve applied the Custom Resource to your cluster, the operator will start reconciling all of Istio’s components.

There are some sample Custom Resource configurations in the config/samples folder. To deploy Istio 1.0.5 with its default configuration options, use the following command:

$ kubectl apply -n istio-system -f config/samples/istio_v1beta1_istio.yaml

After some time, you should see that the Istio pods are running:

$ kubectl get pods -n istio-system --watch NAME READY STATUS RESTARTS AGE istio-citadel-5b69bd4749-h24xk 1/1 Running 0 1m istio-egressgateway-bb8b48cf8-w65hm 1/1 Running 0 1m istio-galley-5ddd798686-jpdlm 1/1 Running 0 1m istio-ingressgateway-678ff4cc87-gkdzt 1/1 Running 0 1m istio-operator-controller-manager-0 2/2 Running 0 9m istio-pilot-fc664fcbd-kgl2k 2/2 Running 0 1m istio-policy-5cf55ff648-xfhf7 2/2 Running 0 1m istio-sidecar-injector-596f8dddbb-gvzk9 1/1 Running 0 1m istio-telemetry-7cbf75c5cf-wk4v8 2/2 Running 0 1m

The Istio Custom Resource is showing Available in its status field and the Istio components are using 1.0.5 images :

$ kubectl describe istio -n istio-system istio Name: istio-sample Namespace: istio-system Labels: controller-tools.k8s.io = 1.0 Annotations: kubectl.kubernetes.io/last-applied-configuration ={ "apiVersion" : "istio.banzaicloud.io/v1beta1" , "kind" : "Istio" , "metadata" : { "annotations" : {} , "labels" : { "controller-tools.k8s.io" : "1.0" } , "name" : "istio-sampl... API Version: istio.banzaicloud.io/v1beta1 Kind: Istio Metadata: Creation Timestamp: 2019-03-31T10:07:22Z Finalizers: istio-operator.finializer.banzaicloud.io Generation: 2 Resource Version: 13101 Self Link: /apis/istio.banzaicloud.io/v1beta1/namespaces/istio-system/istios/istio-sample UID: c6a095da-539c-11e9-9080-42010a9a0136 Spec: Auto Injection Namespaces: default Citadel: Image: istio/citadel:1.0.5 Replica Count: 1 Galley: Image: istio/galley:1.0.5 Replica Count: 1 Gateways: Egress: Max Replicas: 5 Min Replicas: 1 Replica Count: 1 Ingress: Max Replicas: 5 Min Replicas: 1 Replica Count: 1 Include IP Ranges: * Mixer: Image: istio/mixer:1.0.5 Max Replicas: 5 Min Replicas: 1 Replica Count: 1 Mtls: false Pilot: Image: istio/pilot:1.0.5 Max Replicas: 5 Min Replicas: 1 Replica Count: 1 Trace Sampling: 1 Proxy: Image: istio/proxyv2:1.0.5 Sidecar Injector: Image: istio/sidecar_injector:1.0.5 Replica Count: 1 Tracing: Zipkin: Address: zipkin.jaeger-system:9411 Status: Error Message: Status: Available Events: <none>

Deploy sample BookInfo application 🔗︎

Let’s make sure that Istio 1.0.5 is properly installed with Istio’s BookInfo application:

$ kubectl -n default apply -f https://raw.githubusercontent.com/istio/istio/release-1.0/samples/bookinfo/platform/kube/bookinfo.yaml service "details" created deployment.extensions "details-v1" created service "ratings" created deployment.extensions "ratings-v1" created service "reviews" created deployment.extensions "reviews-v1" created deployment.extensions "reviews-v2" created deployment.extensions "reviews-v3" created service "productpage" created deployment.extensions "productpage-v1" created $ kubectl -n default apply -f https://raw.githubusercontent.com/istio/istio/release-1.0/samples/bookinfo/networking/bookinfo-gateway.yaml gateway.networking.istio.io "bookinfo-gateway" created virtualservice.networking.istio.io "bookinfo" created

Determine the external hostname of the ingress gateway and open productpage in a browser:

$ INGRESS_HOST = $( kubectl -n istio-system get service istio-ingressgateway -o jsonpath = '{.status.loadBalancer.ingress[0].ip}' ) $ open http://$INGRESS_HOST/productpage

Install Istio 1.1.1 🔗︎

To install Istio 1.1.1, first we need to check out the release-1.1 branch of our operator (this branch supports the Istio 1.1.x versions):

$ git clone git@github.com:banzaicloud/istio-operator.git $ git checkout release-1.1

If you installed Istio operator with make in the previous section go to to Install Istio Operator with make , if you installed it with helm go to Install Istio Operator with helm . If you haven’t installed Istio operator so far you can choose whichever install option you like.

Install Istio Operator with make

Simply run the following make goal from the project root to install the operator:

$ make deploy

This command will install a Custom Resource Definition in the cluster, and will deploy the operator to the istio-system namespace.

Install Istio Operator with Helm

Alternatively, you can deploy the operator using a Helm chart, which is available in the Banzai Cloud stable Helm repo:

$ helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com $ helm upgrade istio-operator --install --namespace = istio-system --set-string operator.image.tag = 0.1.4 banzaicloud-stable/istio-operator

Note: As of now, the 0.1.4 tag is the latest version of our operator to support Istio versions 1.1.x.

Apply the new Custom Resource

If you’ve installed Istio 1.0.5 or earlier with the Istio operator, and if you check the logs of the operator pod at this point, you will see the following error message: intended Istio version is unsupported by this version of the operator . We need to update the Istio Custom Resource with Istio 1.1’s components for the operator to be reconciled with the Istio control plane.

To deploy Istio 1.1.1 with its default configuration options, use the following command:

$ kubectl apply -n istio-system -f config/samples/istio_v1beta1_istio.yaml

After some time, you should see that new Istio pods are running:

$ kubectl get pods -n istio-system --watch NAME READY STATUS RESTARTS AGE istio-citadel-7664c58768-l8zgb 1/1 Running 0 7m istio-egressgateway-8588c7c8d-wkpgk 1/1 Running 0 7m istio-galley-78b8467b4d-b5dqs 1/1 Running 0 7m istio-ingressgateway-5c48b96cb4-lnfsn 1/1 Running 0 7m istio-operator-controller-manager-0 2/2 Running 0 16m istio-pilot-84588fff4c-4lhq8 2/2 Running 0 7m istio-policy-75f84689f5-78dxr 2/2 Running 0 7m istio-sidecar-injector-66cd99d8c8-bp4j7 1/1 Running 0 7m istio-telemetry-7b667c5fbb-2lfdc 2/2 Running 0 7m

The Istio Custom Resource is showing Available in its status field, and the Istio components are now using 1.1.1 images:

$ kubectl describe istio -n istio-system istio Name: istio-sample Namespace: istio-system Labels: controller-tools.k8s.io = 1.0 Annotations: kubectl.kubernetes.io/last-applied-configuration ={ "apiVersion" : "istio.banzaicloud.io/v1beta1" , "kind" : "Istio" , "metadata" : { "annotations" : {} , "labels" : { "controller-tools.k8s.io" : "1.0" } , "name" : "istio-sampl... API Version: istio.banzaicloud.io/v1beta1 Kind: Istio Metadata: Creation Timestamp: 2019-03-31T10:07:22Z Finalizers: istio-operator.finializer.banzaicloud.io Generation: 3 Resource Version: 21904 Self Link: /apis/istio.banzaicloud.io/v1beta1/namespaces/istio-system/istios/istio-sample UID: c6a095da-539c-11e9-9080-42010a9a0136 Spec: Auto Injection Namespaces: default Citadel: Image: docker.io/istio/citadel:1.1.1 Replica Count: 1 Control Plane Security Enabled: false Default Pod Disruption Budget: Enabled: true Exclude IP Ranges: Galley: Image: docker.io/istio/galley:1.1.1 Replica Count: 1 Gateways: Egress: Max Replicas: 5 Min Replicas: 1 Replica Count: 1 Service Annotations: Service Labels: Service Type: ClusterIP Ingress: Max Replicas: 5 Min Replicas: 1 Replica Count: 1 Service Annotations: Service Labels: Service Type: LoadBalancer K8s ingress: Enabled: false Include IP Ranges: * Mixer: Image: docker.io/istio/mixer:1.1.1 Max Replicas: 5 Min Replicas: 1 Replica Count: 1 Mtls: false Node Agent: Enabled: false Image: docker.io/istio/node-agent-k8s:1.1.1 Outbound Traffic Policy: Mode: ALLOW_ANY Pilot: Image: docker.io/istio/pilot:1.1.1 Max Replicas: 5 Min Replicas: 1 Replica Count: 1 Trace Sampling: 1 Proxy: Enable Core Dump: false Image: docker.io/istio/proxyv2:1.1.1 Proxy Init: Image: docker.io/istio/proxy_init:1.1.1 Sds: Enabled: false Sidecar Injector: Image: docker.io/istio/sidecar_injector:1.1.1 Replica Count: 1 Rewrite App HTTP Probe: true Tracing: Zipkin: Address: zipkin.istio-system:9411 Version: 1.1.1 Status: Error Message: Status: Available Events: <none>

At this point, your Istio control plane is upgraded to Istio 1.1.1 and your BookInfo application should still be available at:

$ open http://$INGRESS_HOST/productpage

Upgrading the Istio control plane between Istio’s major versions with our operator is as easy as deploying a new version of the operator, and then applying a new Custom Resource using the desired component versions.





1. Sidecar upgrades

In order to change sidecars running older versions of the Istio proxy we need to perform a few manual steps. We’re already working on providing a seamless upgrade path through the operator for sidecars.

2. In-place upgrades

An in-place upgrade is impossible right now due to traffic disruptions. It is our plan to support this use-case as well, in the future.

Istio operator - contributing and development 🔗︎

If you’re interested in this project, we happily accept contributions. You can support us by starring the repo, or if you want to help with development, feel free to read our contribution and development guidelines from our first blog post about the operator.

About Banzai Cloud Pipeline 🔗︎

Banzai Cloud’s Pipeline provides a platform for enterprises to develop, deploy, and scale container-based applications. It leverages best-of-breed cloud components, such as Kubernetes, to create a highly productive, yet flexible environment for developers and operations teams alike. Strong security measures — multiple authentication backends, fine-grained authorization, dynamic secret management, automated secure communications between components using TLS, vulnerability scans, static code analysis, CI/CD, and so on — are default features of the Pipeline platform.