Azure DevOps (ADO) Release pipeline

Create Release Pipeline

We will create a new release pipeline. This release pipeline will be linked with the ADO build which we defined earlier, and it will have 2 stages. The first stage creates a new green VMSS (based on the latest VM image created by the last build) and pushes it into the the green backend pool of the Load Balancer. There is a manual approval step post this stage to move into the next stage. Prior to the approval it is possible to verify that the new Green VMSS with the latest application is functioning as expected. On approval the Green VMSS is propagated to handle the production traffic by moving it to the blue backend pool of the load balancer.

Create Release Pipeline

Linking the Release to the Build Artifacts

The release is linked to Build artifact by adding an artifact as shown below.

Linking the Release to the build artifact

Configuring Stage 1 (Which has only one associated task)

The only task in this stage is “Azure Deployment Create or Update Resource Group”. This task deploys the ARM template https://github.com/maniSbindra/vmss-packer-blue-green/blob/master/release-artifacts/vmss-blue-green.json . The first time a Deployment happens using this template it creates the Vnet, Subnet, Public IP Address, Load Balancer (with blue and green backend pools, and rules), and a VMSS based on the latest VM image created in the linked build, and places this VMSS in the Green Backend Pool of the Load Balancer. Since the template deployment mode selected in this task is incremental, the subsequent deployments using this template only create a new VMSS based on the latest VM Image available and the VMSS is placed it in the Green Backend Pool.

Configuring the ARM Template Deployment task

The configuration of this task is shown above. We select the default parameters file (from build artifact) and override the parameters in the “Override template parameters” section using the release pipeline variables. The default parameters file is shown below

{

"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",

"contentVersion": "1.0.0.0",

"parameters": {

"dnsName": {

"value": "javabluegreen1"

},

"adminUsername": {

"value": "azureuser"

},

"adminPassword": {

"value": "thePassword"

},

"greenManagedImageName": {

"value": "newImageId"

},

"greenVMSSName": {

"value": "newimagename"

},

"location": {

"value": "southeastasia"

},

"numberOfVms": {

"value": 1

}

}



}

We override these values in the Override template parameters as follows. The screenshot of the release pipeline variable section is shown later in this section.

-dnsName "$(ARM_DNS_NAME)" -adminUsername "$(VMSS_ADMIN_USER)" -adminPassword "$(VMSS_ADMIN_PASS)" -greenManagedImageName "$(GREEN_VMSS_MANAGED_IMAGE_ID_PREFIX)-$(Build.BuildNumber)" -greenVMSSName "$(Build.BuildNumber)"

The dnsName of the load balancer is fetched from the ARM_DNS_NAME pipeline variable. The username and password for the VMs in the VMSS is fetched from VMSS_ADMIN_USER and VMSS_ADMIN_PASS pipeline variables respectively. The name of the VM image used to create the VMSS is in the same format as the Build, i.e. “$(GREEN_VMSS_MANAGED_IMAGE_ID_PREFIX)-$(Build.BuildNumber)” for build number 20190315.2 corresponds to VM image name simpleapp-20190315.2. Similarly the name of the Green VMSS is same as the build number which would be something like 20190315.2

Setting up post deployment approval after Stage 1

Once Stage 1 is complete, and pending approval, the team can test the new version of the application by hitting port 8080 on the loadbalancer (which the load balancer rule sends to the VMSS in the Green backend pool on port 8080). Once the verification is done the team can propagate the green VMSS to production by manually approving this stage. This approval is configured in the release gate as shown below

Configuring Stage 2 (Propagate Green VMSS to handle production traffic)

This stage is used to propagate the green VMSS into production. During the first run of this release there is no VMSS in the Blue backend pool of the load balancer, and the Green VMSS can be added to the empty blue backend pool. From the second execution onwards we first need to remove the VMSS with the old version of the app out of the blue backend pool and then move the green VMSS with the new version of the app in the blue backend pool. For this reason we need to maintain the name of the VMSS in the blue backend pool in the release pipeline itself. We store the blue VMSS name in the release pipeline variable “BLUE_VMSS_NAME”. Let us first look at the configuration of the first task.

Configure the Agent

As earlier we select the Hosted Ubuntu 1604 agent pool. In this stage since we need to write the name of the VMSS propagated to the blue backend pool to the release variable BLUE_VMSS_NAME we need to give the agent access to the OAuth token (as shown in the screenshot below), to enable it to edit the release definition and update the release variable. We also need to give the user “Project Collection Build Service” user access to edit the release pipeline. The answer in the stack overflow post gives more details on this configuration.

Configure Agent

Configure the Azure CLI task (Which executes the VMSS propagation shell script)

We first select the script to be executed by the Azure CLI task as release-artifacts/swap-commands.sh, and the arguments to pass to this script.

Configure the Azure CLI task

The script does 3 things. If the BLUE_VMSS_NAME exists then it is removed from the blue backend pool. Next the GREEN_VMSS_NAME is removed from Green Backend Pool and finally the GREEN_VMSS_NAME is added to the blue backend pool. The script accepts 4 parameters, the BLUE_VMSS_NAME, the GREEN_VMSS_NAME, the RESOURCE_GROUP where the VMSS’s exist, and the LOAD_BALANCER_BLUE_BACKEND_POOL_ID, so that the GREEN_VMSS can be added to the blue backend pool.

#!/bin/bash



echo "************* Setting Variables from opts"



while getopts :r:g:p:b:h opt; do

case "$opt" in

r)

RESOURCE_GROUP="$OPTARG"

;;

g)

GREEN_VMSS_NAME="$OPTARG"

;;



p)

LOAD_BALANCER_BLUE_BACKEND_POOL_ID="$OPTARG"

;;

b)

BLUE_VMSS_NAME="$OPTARG"

;;

h)

echo "todo add help"

;;

esac

done



set -x



echo "RESOURCE_GROUP: $RESOURCE_GROUP"

echo "BLUE_VMSS_NAME: $BLUE_VMSS_NAME"

echo "GREEN_VMSS_NAME: $GREEN_VMSS_NAME"

echo "LOAD_BALANCER_BLUE_BACKEND_POOL_ID: $LOAD_BALANCER_BLUE_BACKEND_POOL_ID"



## IF BLUE VMSS EXISTS REMOVE IT FROM BLUE BACKEND POOL

# ------------------------------------------------------

if [ ! -z "$BLUE_VMSS_NAME" ]

then



echo "Removing Blue VMSS from blue backend pool ..."



# Remove active blue vmss from blue backend pool

az vmss update -g $RESOURCE_GROUP -n $BLUE_VMSS_NAME --remove virtualMachineProfile.networkProfile.networkInterfaceConfigurations[0].ipConfigurations[0].loadBalancerBackendAddressPools 0



echo "Blue VMSS removed from blue backend pool"

fi

# ------------------------------------------------------







## REMOVE GREEN VMSS FROM GREEN BACKEND POOL

# ------------------------------------------------------

echo "Removing Green VMSS from Green backend pool ..."



# Remove new green vmss from green backend pool

az vmss update -g $RESOURCE_GROUP -n $GREEN_VMSS_NAME --remove virtualMachineProfile.networkProfile.networkInterfaceConfigurations[0].ipConfigurations[0].loadBalancerBackendAddressPools 0



# Remove new green vmss from green nat pool

az vmss update -g $RESOURCE_GROUP -n $GREEN_VMSS_NAME --remove virtualMachineProfile.networkProfile.networkInterfaceConfigurations[0].ipConfigurations[0].loadBalancerInboundNatPools 0



echo "Green VMSS removed from Green backend pool"

# ------------------------------------------------------





# ADD GREEN VMSS TO BLUE BACKEND POOL

# ------------------------------------------------------

echo "Adding Green VMSS to backend pool ..."



# Add new green vmss to blue backend pool

az vmss update -g "$RESOURCE_GROUP" -n "$GREEN_VMSS_NAME" --add virtualMachineProfile.networkProfile.networkInterfaceConfigurations[0].ipConfigurations[0].loadBalancerBackendAddressPools "{\"id\": \"$LOAD_BALANCER_BLUE_BACKEND_POOL_ID\"}"

# ------------------------------------------------------

The arguments are passed to the scripts using the release pipeline variables as follows:

-r $(RESOURCE_GROUP) -g "$(Build.BuildNumber)" -p "$(LOAD_BALANCER_BLUE_BACKEND_POOL_ID)" -b "$(BLUE_VMSS_NAME)"

Let us look at the pipeline variable configuration

The LOAD_BALANCER_BLUE_BACKEND_POOL_ID is set in the format /subscriptions/{your-subscription-id}/resourceGroups/vmss-blue-green-rg/providers/Microsoft.Network/loadBalancers/lb-vmss-ado-blue-green/backendAddressPools/lbBEBlue

Release pipeline Variable configuration

Configuration of the powershell script to add name of new VMSS (which has now been moved to the blue backend pool) to the variable BLUE_VMSS_NAME

The powershell script is similar to the one shown in the stackoverflow link above.