Microsoft Azure Pipelines is a new tool for building Continuous Delivery (CD) pipelines, a part of the Azure DevOps – an end-to-end service to track and share code, and ship solutions. Is it a SaaS product, based on Microsoft Visual Studio Team Services (VSTS), offering cloud-hosted pipelines for Linux, macOS, and Windows.



Build Options

Azure Pipelines supports builds on a wide selection of hosts, including Linux, Windows and MacOS. Azure Pipelines also supports “container builds” where it pulls specified images from a public repository and uses a container running off that image to execute build steps.

Microsoft gives away free Azure Pipelines tier subscriptions with one container running a job that takes up to 30 minutes for commercial products and/or open source software not hosted on Azure. For Open Source software repositories hosted on Azure, Microsoft offers a free pool of 10 build hosts and permits jobs running up to six hours, before they would auto-cancel.

Pipelines may be coded using YAML or constructed through a visual designer. Azure Pipelines provides utilities for automatically installing desired development tools for most common software stacks, including: Node.JS, Go, C#/.NET, Xcode and others.



Parallel Builds

If the build process is parallelizable – we could start some of the steps without waiting for the result of the previous operation, Azure allows us to run several jobs at the same time. In order to execute multiple jobs in parallel, we could utilize the phases functionality. For example, if we want to build a project for Linux, Windows and MacOS, we could define a phase for each Operating System type. Our definition might look like the below:

phases: ################################################# - phase: windows ################################################# condition: succeeded() queue: Hosted VS2017 ################################################################################ steps: ... 1 2 3 4 5 6 7 8 9 phases : ################################################# - phase : windows ################################################# condition : succeeded ( ) queue : Hosted VS2017 ################################################################################ steps : . . .

Please note the word condition. We could specify there conditions such as succeeded() and failed() as compare operations, for example with pipeline variables, as well as logical operations. In our instance, if a job failed, our phase will skip execution.

We could add three (3) pipeline variables skip_windows, skip_linux and skip_darwin. When one of those is set to true, we skip the phase associated with the corresponding OS. In order to do this, we will extend the condition to:

condition: and(succeeded(), ne(variables.skip_windows, true)) 1 condition : and ( succeeded ( ) , ne ( variables . skip_windows , true ) )

Please note that when a pipeline variable is used in a condition it is referred to as variables.VAR.



Pipeline Definition

The main part of the pipeline are the build steps. For example, below is a part of the Node.JS pipeline:

- script: npm config set package-lock false # install - task: Npm@1 displayName: npm install inputs: command: install # lint - script: npm run ci-lint displayName: npm lint # test - script: npm run ci-test displayName: npm test 1 2 3 4 5 6 7 8 9 10 11 12 - script : npm config set package - lock false # install - task : Npm @ 1 displayName : npm install inputs : command : install # lint - script : npm run ci - lint displayName : npm lint # test - script : npm run ci - test displayName : npm test



The task step is a kind that uses Azure Pipelines tools. In this instance, our install step uses an Azure tool designed for Node.JS. The script -type step executes commands directly on the build agent shell.

To set what branch or branches, CD platform should monitor for changes, we use the trigger property.

# run our pipeline on changes to the below branch[es] trigger: - master 1 2 3 # run our pipeline on changes to the below branch[es] trigger : - master



Container Builds

Container builds are currently an experimental feature, recently introduced to Azure Pipelines. In order to use this functionality, we configure YAML as below:

resources: containers: - container: vstsbuild image: diophant/golang_vsts:1.0.0 jobs: - job: Linux pool: vmImage: Ubuntu 16.04 container: vstsbuild steps: … 1 2 3 4 5 6 7 8 9 10 11 12 13 14 resources : containers : - container : vstsbuild image : diophant / golang_vsts : 1.0.0 jobs : - job : Linux pool : vmImage : Ubuntu 16.04 container : vstsbuild steps : …

Presently, pool vmImage has to be set to “Ubuntu 16.04” for the container build to work. Our build is being done using the container diophant/golang_vsts:1.0.0, and host Ubuntu 16.04. It is important not to invoke the USER command inside the containers used for Azure Pipelines builds to work. The Azure software itself wants, as root, to create a new user with the sudo rights.



Environmental Variables

Our Node.JS project build is configured to support multiple base OS images and Node.JS versions. This configuration is passed to the build pipeline steps as environmental variables.

In order to use environmental variables, on the Azure DevOps platform, as a part of conditions, use the below syntax.

condition: and(succeeded(), ne(variables.skip_windows, true)) 1 condition : and ( succeeded ( ) , ne ( variables . skip_windows , true ) )

In the condition section, an environmental variable skip_windows has to have variables in front of it.

If we need environmental variables to be accessible from inside tasks, such as “script,” we use the below syntax.

- script: | if [[ -z $OS || -z $VERSION ]]; then echo 'one or more variables are undefined, assuming script debug mode' export VERSION=10.0.0 export OS=centos7 else source contrib/etc/util.sh if osSupported ${OS}; then export FROM_DATA=\"$(getBaseImageForOs)\" else echo "Unsupported os: $OS" exit 1 fi fi echo "##vso[task.setvariable variable=FROM_DATA]$FROM_DATA" echo "##vso[task.setvariable variable=OS]$OS" echo "##vso[task.setvariable variable=VERSION]$VERSION" displayName: Configure env: OS: $(OS) VERSION: $(VERSION) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 - script : | if [ [ - z $ OS || - z $ VERSION ] ] ; then echo 'one or more variables are undefined, assuming script debug mode' export VERSION = 10.0.0 export OS = centos7 else source contrib / etc / util . sh if osSupported $ { OS } ; then export FROM_DATA = \ " $ ( getBaseImageForOs ) \ " else echo "Unsupported os: $OS" exit 1 fi fi echo "##vso[task.setvariable variable=FROM_DATA]$FROM_DATA" echo "##vso[task.setvariable variable=OS]$OS" echo "##vso[task.setvariable variable=VERSION]$VERSION" displayName : Configure env : OS : $ ( OS ) VERSION : $ ( VERSION )

Above, in the env section, we pass variables OS and VERSION to the “script” task Configure. We also, inside this task, persist multiple environmental variables to be available for the subsequent steps using the below syntax.

echo "##vso[task.setvariable variable=OS]$OS" 1 echo "##vso[task.setvariable variable=OS]$OS"

The above line, internally, generates a Write-Host command persisting variable OS, as defined by variable=OS and value of the persisted OS is $(OS) – the current value of OS.



Triggering Build

Azure Pipelines uses webhooks to watch code repositories for updates to specified branches and pull requests, automatically triggering the build pipeline when the set conditions are met.

Sometimes, just watching repository is not enough. We design Cloud-Native applications where individual components packaged as containers. Let’s say that there is a new security advisory against the base image that we used. Hopefully, the maintainers of base centos or alpine images affected by the advisory updated their official OS images with a version remediating the problem, shortly after an advisory was published. What could be done about already released images, which include the affected base layers?

DevOps teams cloud and should deploy tools such as Aqua and Snyk to monitor their packages and containers to sanitize their solutions. This, however, still leaves IT teams with the need to manually rebuild all affected container images.

We developed an in-house solution using stateless functions such as AWS Lambda or Azure Functions to trigger builds, updating final docker images, whenever the base OS image is updated. The “watcher” Lambda function is triggered by the AWS CloudWatch timer, and it triggers the function checking whether any images have to be rebuilt. If necessary, this function calls the trigger function that uses REST API to request builds from the DevOps endpoints for outdated/missing releases. This is just one example of IT automation procedures that DevOps engineers could deploy.



Azure REST API

Microsoft Azure Pipelines offer a rich set of interfaces for automating DevOps. There are multiple ways to authenticate with Azure Pipelines. We used the Personal Access Token (PAT), which could be easily created using Azure Pipelines web interface. To make the PAT suitable for use with REST API calls, we need to prefix the token with a username of choice (not validated) and a colon, then BASE-64 encode the resulting string and use in a header ‘Authentification’ as ‘Basic {tokenstring}’.

{ "definition":{ "id":"1" }, "parameters":"{ \"VERSION\":\"10.12.0\", \"OS\":\"alpine3\" }" } 1 2 3 4 5 6 7 8 9 { "definition" : { "id" : "1" } , "parameters" : "{ \"VERSION\":\"10.12.0\", \"OS\":\"alpine3\" }" }

The id parameter is the Azure Pipelines build id. The parameters value has to be a string with escaped JSON object, containing those pipeline variables that want to be set at the time the build is being triggered.

Despite being a relatively new DevOps tool, Azure Pipelines is a powerful and flexible SaaS platform for running CD workloads, for both Open Source and Commercial projects.