Kubernetes is a Production grade container orchestration platform with automated scaling and management of containerized applications. It is also open source, so you can install Kubernetes on any cloud like AWS, Digital Ocean, Google Cloud Platform, or even just on your own machines on premises. Kubernetes was started at Google and is also offered as a hosted Container Service called GKE. With Shippable, you can easily hook up your automated DevOps pipeline from source control to deploy to your Kubernetes pods and accelerate innovation.

In this blog, we demonstrate how to deploy a load balanced, multi-container application to multiple Kubernetes environments on GKE. The deployment occurs in multiple stages in a Shippable defined workflow.

Get Started: Kubernetes Deployment spec

The pods and services (load balancer) for the application are created using a deployment spec. Instead of creating and maintaining a deployment spec per environment which is a common practice, we create a single deployment spec template. This template has placeholders for the image and service/pod labels. When we deploy the application to a specific environment, we use powerful yet simple Shippable platform functions and resources to replace these placeholders at run time when the deployment actually happens.

The deployment spec template (located here in our public repository) defines the label selectors placeholders in the .spec.selector section and the labels for the pods in the .spec.template.metadata.labels section. Labels are defined for both the front end voting application (FE_LABEL) as well as the Redis service (BE_LABEL) which the front ends makes API calls on via another load balancer.

Application architecture

The application is a simple voting web app with the web-tier authored in Python / Flask. The front-end runs its own pod behind a front end load balancer. The votes are stored in a Redis cache, which runs in another pod. The Redis container runs behind another backend load balancer. Our deplyoment workflow thus creates two load balancers and two pods, each of which can have its own set of replicas.

Multi-stage deployment workflow

In our multi-stage deployment scenario, we deploy the application in several stages.

Stage 1 : Stage 1 runs on every commit to the application source code, builds the docker image and pushes it to GCR.

Stage 2 : Stage 2 runs on successful completion of Stage 1. In Stage 2, we deploy the application to a test environment on the cluster using labels and create a load balancer for the application.

Stage 3: In Stage 3, we deploy the application to a production environment on the cluster using labels and create a load balancer for the application. Stage 3 is configured to run manually.

This is a pictorial representation of the workflow we're going to configure. The green boxes are jobs and the grey boxes are the input resources for the jobs. The workflow is defined across two configuration files: shippable.jobs.yml and shippable.resources.yml.

build_image is a runSh job that builds the voting app image called front_img and pushes the image to GCR. It executes Stage 1 of our workflow.

is a runSh job that builds the voting app image and pushes the image to GCR. It executes of our workflow. deploy_test is a runSh job that takes the deployment template, replaces the image and labels placeholders in the template with the actual values from the front_img and test_params resources, and deploys both front_img and redis images . It also outputs relstate which is an updated deployment spec with immutable image tags, used in Stage 3. It executes Stage 2 of our workflow.

is a runSh job that takes the deployment template, replaces the image and labels placeholders in the template with the actual values from the resources, and deploys both . It also outputs which is an updated deployment spec with immutable image tags, used in Stage 3. deploy_prod is a runSh job that takes the deployment spec from relstate , replaces the labels needed for prod deployment using the prod_params resource, and deploys both frontend_img and redis images to production. It executes Stage 3 of our workflow.

is a runSh job that takes the deployment spec from , replaces the labels needed for prod deployment using the resource, and deploys both and images to production. These jobs have input and output resources that are described in detail in other sections below.

Prerequisites

If you're not familiar with Shippable, it is also recommended that you read the Platform overview doc to understand the overall structure of Shippable's DevOps Assembly Lines platform.

Sample project

The code for this example is in GitHub at devops-recipes/ship-voting-app-redis. You can fork the repository to try out this sample yourself or just follow instructions to configure your own use case

The voting app source code and Dockerfile can be found here in the repository.

This repository also has the Shippable configuration files to setup the multi-stage pipeline .

ship_vote_all_in_one_redis.yml.template used to create the pods and services can be found in the repository. The Kubernetes deployment specused to create the pods and services can be found here

1. Build application Docker image

A. Add the subscription integration for your SCM account integration

By default, you will already have an account integration with whichever SCM provider you've used to log into Shippable. If your source code repository is on that SCM account, you should use it as is. You might have to add it to your Subscription by following the steps here: Adding an account integration to a Subscription.

If your source code repository is on another SCM account, create an account integration for it first by using one of the following supported SCM docs:

B. Create account integration for Google Cloud.

Since your deployment config will interact with GCR to push the front-end image and GKE to deploy the application, you will need to create an integration for Google Cloud in the Shippable UI.

Create a Google Cloud integration called drship_gcloud using instructions found here.

using instructions found here. Ensure that you have set the Subscription Scopes in the account integration to the subscription where your respository is located.



C. Define resources used to build the application Docker image

Add the following to your shippable.resources.yml file:

resources : - name : ship_voting_app_redis_gitRepo type : gitRepo # replace dr_github with your GitHub integration name integration : dr_github pointer : # replace with source code location (e.g. GitHub) where you cloned this # sample project. sourceName : devops-recipes/ship-voting-app-redis branch : master - name : gcloud_cliConfig type : cliConfig integration : drship_gcloud pointer : # replace us-central1-a with your availability zone region : us-central1-a - name : front_img type : image integration : drship_gcloud pointer : # replace devops-samples with your google cloud project name sourceName : " gcr . io / devops - samples / vote " seed : versionName : " master.1 "

ship_voting_app_redis_gitRepo is a is a gitRepo resource which is a pointer to the git repository that contains the source code for this application.

gke_cliConfig is a cliConfig resource which is a pointer to the private key of your GKE service account needed to initialize the gcloud CLI.

cliConfig front_img is an image resource that represent the Docker image of the vote application. This uses the Google Cloud integration that we created earlier.

D. Define build_image

Add the build_image job to your shippable.jobs.yml file.

It is a runSh job that lets you run any shell script. It takes app_gitRepo as an input, builds the voting app image and pushes the docker image to GCR.

It sets the image tag in the front_img resource and outputs that resource so that a new version of the resource can be created. This in turn triggers Stage 2 of our pipeline since front_img is an input to the Stage 2 job.

jobs : - name : build_image type : runSh steps : - IN : ship_voting_app_redis_gitRepo - IN : gcloud_cliConfig - OUT : front_img - TASK : # build the ship-vote image and push it to GCR - script : |

pushd $(shipctl get_resource_state "ship_voting_app_redis_gitRepo")/ship-vote export IMG_TAG=master.$BUILD_NUMBER



# Replace devops-samples with your google cloud project name export IMG_REF=gcr.io/devops-samples/vote:$IMG_TAG docker build -t $IMG_REF . gcloud docker -- push $IMG_REF # Persist the image tag in the front_img resource so that the next stage # of the workflow knows which tag to deploy shipctl put_resource_state front_img "versionName" "$IMG_TAG" popd

E. Commit config files and add them to your Shippable account.

Once you have these configuration files as described above, commit them to your repository. The shippable.jobs.yml and shippable.resources.yml can be committed to the same app repository, or to a separate repository.

The repository containing your jobs and resources ymls is called a Sync repository and basically represents your workflow configuration.

Follow these instructions to import your configuration files into your Shippable account.

Stage 1 is complete at this point

2. Deploy application to Test

A. Define resources used in the second stage

Add the following to your shippable.resources.yml file:

resources : - name : test_params type : params version : params : FE_LABEL : " vote-front-test " BE_LABEL : " vote-back-test "

- name : relstate type : state - name : teststate type : state - name: kube_cluster

type: cluster

integration: drship_gcloud

pointer:

# replace devops-test-cluster with your kubernetes cluster name

sourceName: "devops-test-cluster"

# replace us-central1-a with your availability zone

region: us-central1-a

t est_params is a params resource that specifies the labels for the load balancer service and the pods that run our images.

params relstate and teststate are state resources where we store shared state. relstate stores the deployment spec template that is used in Stage 3. teststate stores the deployment spec that was used to create the pods and services in Stage 2 for the test environment.

kube_cluster is a cluster resource that points to your Kubernetes cluster on GKE.

B. Define deploy_test

Add the deploy_test job to your shippable.jobs.yml file.

It is a runSh job that takes the deployment template, replaces the image and labels placeholders in the template with the actual values from the front_img and test_params resources, and deploys both front_img and redis images.

It also outputs relstate which is an updated deployment spec with immutable image tags, used in Stage 3. It executes Stage 2 of our workflow.

jobs : - name : deploy_test type : runSh steps : - IN : ship_voting_app_redis_gitRepo switch : off - IN : gcloud_cliConfig - IN : front_img - IN : test_params - IN : kube_cluster - OUT : relstate - OUT : teststate - TASK : # Create a Kubernetes deploy spec and service for the test environment for an existing cluster and deploy them - script : | pushd $(shipctl get_resource_state "ship_voting_app_redis_gitRepo")/kubernetes-manifests # Get the image tag from the front_img resource that we persisted earlier export FE_TAG=$(shipctl get_resource_version_name "front_img") # Save the image tag in the shared state resource relstate # so that we can also use it in the next stage for prod deployment. shipctl post_resource_state "relstate" "FE_TAG" $FE_TAG # Save the template file in the shared state resource relstate # so that we can also use it in the next stage for prod deployment. cp ./ship_vote_all_in_one_redis.yml.template ./release.template.yml shipctl refresh_file_to_out_path ./release.template.yml relstate # Replace the image and label placeholder values in our template cp ./ship_vote_all_in_one_redis.yml.template ./test_deploy_spec.yml shipctl replace ./test_deploy_spec.yml shipctl refresh_file_to_out_path ./test_deploy_spec.yml teststate # Init the kubeconfig file for the cluster since we are going to use kubectl gcloud container clusters get-credentials $KUBE_CLUSTER_POINTER_SOURCENAME --zone $KUBE_CLUSTER_POINTER_REGION # Delete previous deployments and services kubectl delete deployment $FE_LABEL 2>/dev/null || echo "" kubectl delete deployment $BE_LABEL 2>/dev/null || echo "" kubectl delete service $FE_LABEL 2>/dev/null || echo "" kubectl delete service $BE_LABEL 2>/dev/null || echo "" # Create the service and deployment object in the test environment kubectl create -o json -f ./test_deploy_spec.yml > kube_output.json cat kube_output.json popd

C. Commit the shippable.resources.yml and shippable.jobs.yml file to your repository. This completes Stage 2 of our workflow.

3. Deploy application to Prod

A. Define resources used in the third stage

Add the following to your shippable.resources.yml file:

resources : - name : prodstate type : state

prodstate is a state resource that stores the deployment spec that was used to create the pods and services in Stage 3 for the prod environment.

B. Define deploy_prod

Add the deploy_prod job to your shippable.jobs.yml file.

It is a runSh job that takes the deployment template, replaces the labels placeholders in the template with the actual values from the prod_params resource. It deploys both front_img and redis images to the prod environment and also outputs prodstate. It executes Stage 3 of our workflow.

jobs : - name : deploy_prod type : runSh steps : - IN : deploy_test switch : off - IN : gke_cliConfig - IN : prod_params - IN : kube_cluster - IN : relstate - OUT : prodstate - TASK : # Create a Kubernetes deploy spec and service for the Prod environment for an existing cluster and deploy them - script : | # Extract the image tag from the previous stage of the workflow export FE_TAG=$(eval echo "$"$(shipctl get_resource_version_key "relstate" "FE_TAG")) # Get the deployment spec template from relstate and replace the # image and label placeholder values. shipctl copy_resource_file_from_state relstate release.template.yml . cp ./release.template.yml ./prod_deploy_spec.yml shipctl replace ./prod_deploy_spec.yml # Copy the final deployment spec to prodstate shipctl refresh_file_to_out_path ./prod_deploy_spec.yml prodstate # Init the kubeconfig file for the cluster since we are going to use kubectl gcloud container clusters get-credentials $KUBE_CLUSTER_POINTER_SOURCENAME --zone $KUBE_CLUSTER_POINTER_REGION # Delete previous deployments and services kubectl delete deployment $FE_LABEL 2>/dev/null || echo "" kubectl delete deployment $BE_LABEL 2>/dev/null || echo "" kubectl delete service $FE_LABEL 2>/dev/null || echo "" kubectl delete service $BE_LABEL 2>/dev/null || echo "" # Create the service and deployment object in the prod environment # using the prod deployment spec kubectl create -o json -f ./prod_deploy_spec.yml > kube_output.json cat kube_output.json

C. Commit the shippable.resources.yml and shippable.jobs.yml file to your repository. This completes Stage 2 of our workflow.

Your pipeline should now look like this in the SPOG view

4. Trigger your pipeline

To test the pipeline, commit a change to the voting app. This will trigger the first two stages of the workflow automatically after which you can manually trigger the third stage of the workflow.

Grid view of all the jobs

Stage 1 screenshot

Stage 2 screenshot

Stage 3 screenshot