Imagine your project consists of 4 microservices (3 backends, 1 frontend). Yesterday you introduced several new features and made a release. Unfortunately, your users have just reported a bug. Some of the old important features are not working. You need to do a rollback for all services. Ah, if it could be done with one button.

Actually it can be. In this article I’ll show you how.

Tech stack:

Jenkins for rollback automation

Ansible + Python for rollback script

Docker registry for storing release images

DC/OS for running apps

Overview

We will have a python script, called via Ansible from Jenkins, as described in this article. The only difference is — we should have two different tags to run. The first one gathers all available images, the second runs the rollback.

The get algorithm:

Request all images from the docker registry. Filter them by environment, sort by date and take 10 last one for every repository. Form json with repositories, images and dates and write to file system

The run algorithm:

Read json from get second step and create a Jenkins input Take all available images for the selected date and do a rollback

The rollback itself:

Modify the docker image section in marathon json config Start a deploy with modified config

Special case

Imagine a service, which doesn’t change in this release. It means there won’t be any rollback image available for it. But you still need to roll it back, because of the compatibility issues. Please find the example of the situation on the picture below.

If you select Today-1 only Repo1 and Repo3 will be rolled back, as there are no images for Repo2 . Perhaps it wasn't changed.

The problem here is that Repo1 or Repo3 N-1 versions could be incompatible with Repo2 latest version. So you need to find the next version of Repo2 before the rollback date. It is Today-2 version.

Get rollbacks

We will have two actions for a rollback:

We gather all rollback dates and images available for the current environment.

User selects the data and we perform a rollback.

Ansible side

Ansible changes are minor. Just add two tags for common steps (like requirements installation):

- name: "Copy requirements.txt"

copy:

src: "requirements.txt"

dest: "/tmp/{{ role_name }}/"

tags:

- get

- run

Don’t forget to add tag to the always step, or your clean-up will be ignored. Using run tag only is preferred.

It would be useful to register rollbacks in get output and debug them. In this case you can use Ansible even without Jenkins.

- name: "Get rollbacks"

shell: "source activate /tmp/{{ role_name }}/{{ conda_env }} ; {{ item }}"

with_items:

- pip install -r /tmp/{{ role_name }}/requirements.txt

- "python /tmp/{{ role_name }}/rollback.py get --repo={{ repo }} --dump={{ dump_path }} --env={{ env }}"

args:

executable: /bin/bash

tags:

- get

register: rollbacks - debug:

var: rollbacks.results[1].stdout

tags:

- get

Python side

With docopt you can use a single entry point with two options, one for get and one for run .

Usage:

rollback.py get --repo=<r> --env=<e> [--dump=<dr>]

rollback.py run --date=<d> --env=<e> --slack=<s> --user=<u> --pwd=<p> [--dump=<dr>]

The fork itself:

if arguments['get']:

return get(repo, env, dump)

if arguments['run']:

return run(date, env, slack, user, pwd, dump)

To get rollbacks you need to call you Docker registry’s API first.

I assume that you use this image naming schema:

<private-docker-registry-host:port>/service-name:build-number-branch

You need to get all tags for current repo, filter them by environment, then sort by date and return last 10.

def get_rollbacks(repo: str, env: str):

r = requests.get(f'{DOCKER_REGISTRY}/v2/{repo}/tags/list', verify=False)

if r.status_code != 200:

raise Exception(f"Failed to fetch tags {r.status_code}")

releases = list(filter(lambda x: x.endswith(env), r.json()['tags']))

all_rollbacks = [(get_manifest(repo, r), {r: repo}) for r in releases[-10:]]

return dict(all_rollbacks)

Where repo is your service-name and env is the current branch.

Sorting by date is a bit complex. Date is not included in tags information. The only way to get it is to fetch the mainfest and to check history .

def get_manifest(repo, tag):

r = requests.get(f'{DOCKER_REGISTRY}/v2/{repo}/manifests/{tag}', verify=False)

if r.status_code != 200:

raise Exception(f"Failed to fetch manifest {r.raw}")

history = r.json()['history']

sort = sorted([json.loads(h['v1Compatibility'])['created'] for h in history])

return sort[-1][:10]

The full get function:

def get(repo: str, env: str, dump: str):

rollbacks = {}

repos = repo.split(',')

for r in repos:

for date, rb in get_rollbacks(r, env).items():

if date in rollbacks:

rollbacks[date] += [rb]

else:

rollbacks[date] = [rb]

print(rollbacks)

if dump is not None:

with open(path.join(dump, "rollback.json"), mode='w') as rb:

json.dump({'all': repos, 'rollbacks': rollbacks}, rb) return

rollbacks.keys()

Where repo is a comma separated list of your service-names. F.e. repo1,repo2,repo3 . You also need to print rollbacks for Ansible debug.

Jenkins side

Let’s start Jenkins pipeline with environment input.

parameters {

choice(choices: 'dev

stage

prod', description: 'Which environment should I rollback?', name: 'environment')

}

if you use master environment instead of prod you don’t need to do anything. Otherwise you need to create static variable rollback_env outside of the pipeline and fill it during the first step.

script {

// need this as env names don't match each other. develop/master/stage in docker vs dev/stage/prod in marathon

if (params.environment == 'prod') {

rollback_env = "master"

} else if(params.environment == 'stage') {

rollback_env = "stage"

} else {

rollback_env = "develop"

}

}

Then just download your git repo with ansible playbook and run it.

git branch: 'master',

credentialsId: <your git user credentials id>',

url: "<your ansible repo>"

ansiblePlaybook(

playbook: "${env.PLAYBOOK_ROOT}/rollback_service.yaml",

inventory: "inventories/dev/hosts.ini",

credentialsId: <your git user credentials id>',

extras: '-e "repo=' + "${env.REPOS}" + ' env=' + "${docker_env}" + ' slack=' + "${env.SLACK_CALLBACK}" + ' dump_path=' + "/tmp" + '" -v',

tags: "get")

Please pay attention to the dump_path . It tells python script to create json directly in the /tmp , so that we can read it from Jenkins. Lets do it.

import groovy.json.JsonSlurper def gather_rollback_dates() {

def inputFile = readFile("/tmp/rollback.json")

def InputJSON = new JsonSlurper().parseText(inputFile)

return InputJSON['rollbacks'].keySet().join("

")

}

This function will find your rollback, get all dates and form a string with

separator. It is required to generate an input with dropdown.

stage('Select rollback date') {

steps {

script {

def userInput = false

try {

timeout(time: 120, unit: 'SECONDS') {

userInput = input(id: 'userInput',

message: 'Select a date to rollback',

parameters: [

choice(name: 'rollback_date',

choices: gather_rollback_dates(),

description: 'One or more services have rollback at this date')])

}

} catch(err) {

}

if (userInput) {

print('Performing rollback')

env.DATE = userInput

} else {

print('Skip rollback')

}

}

}

}

It looks like this:

Perform a rollback

We have 5 actions for a rollback:

Read json from previous step

Find missing images for the selected date

Get marathon service ids from docker ids

Change marathon app’s config

Update app in marathon

Ansible side

Nothing special here. Just call python.

- name: "Perform rollbacks"

shell: "source activate /tmp/{{ role_name }}/{{ conda_env }} ; {{ item }}"

with_items:

- pip install -r /tmp/{{ role_name }}/requirements.txt

- "python /tmp/{{ role_name }}/rollback.py run

--date={{ date }}

--env={{ env }}

--slack={{ slack }}

--user={{ dcos_user }}

--dump={{ dump_path }}

--pwd={{ dcos_password }}"

tags:

- run

Python side

Let’s start with run method

Read json and select all available images for a selected date.

def run(date, env, slack, user, pwd, dump):

json_data = read_rollbacks(dump)

all_rollbacks = OrderedDict(sorted(json_data['rollbacks'].items(), key=lambda x: x[0]))

repos = json_data['all']

images = all_rollbacks[date]

If images for some repos are missing — we need to find their older versions. Add this to your run method:

if len(repos) > 1 and len(repos) > len(images):

get_missing_images(date, repos, all_rollbacks)

Where get_missing_images just goes through all_rollbacks and selects image with nearest date for each missing image.

def get_missing_images(date, repos, all_rollbacks):

images = all_rollbacks[date] # select available images

found_services = [list(rb.values())[0] for rb in images] # get services from images

missing = list(set(repos) - set(found_services)) # substract to get missing

for service in missing: # populate images with rollback for every missing

rollback = get_nearest_date(service, date, all_rollbacks)

if rollback is None:

print(f"Previous rollback for {service} not found")

else:

images += [rollback] def get_nearest_date(repo, date, all_rollbacks):

for d, images in reversed(all_rollbacks.items()):

if d < date:

for rollback, image in images[0].items():

if image == repo:

return {rollback: image}

return None

After we have our images populated we need to get marathon service ids. Our marathon ids uses standard /<department>/<environment>/<project>/<service-name> . At this step we have only service-name , so we should create a binding to Maration id.

We can do it by listing all applications running in Maration and filtering them by the environment and service name (I haven’t found better solution).

def get_service_ids(env: str, images: list, user: str, pwd: str) -> dict:

ids_only = get_marathon_ids_for_env(env, user, pwd) # all running services for env

services = {}

for rollback in images:

tag = list(rollback.keys())[0]

id_part = rollback[tag]

real_id = list(filter(lambda x: x.endswith(id_part), ids_only)) # filter by service-name

if not real_id:

raise Exception(f"Id {id_part} not found")

services[real_id[0]] = tag

return services def get_marathon_ids_for_env(env: str, user: str, pwd: str):

res = call_with_output(f'dcos auth login --username={user} --password={pwd}')

if res.decode().strip() != 'Login successful!':

raise Exception("Can't login to dcos cli")

all_services = call_with_output('dcos marathon app list')

matched = list(filter(lambda x: x.startswith(f"/ds/{env}"), all_services.decode().split('

')))

return [m.split(' ')[0] for m in matched]

After we have service ids we can iterate through them and do a rollback for each. Add this to your run method:

services = get_service_ids(env, images, user, pwd)

for service_id, service_tag in services.items():

if slack is not None:

notify_slack(slack, f"Rollback { service_id }: { service_tag }")

print(do_deploy(service_id, service_tag))

Well, that’s all. Don’t forget to add slack notifications for the rollback.

Jenkins side

Python part was the most complex. On Jenkins side you just need to call Ansible with run tag and selected date.

stage('Rollback') {

when {

expression {

return env.DATE != null

}

}

steps {

ansiblePlaybook(

playbook: "${env.PLAYBOOK_ROOT}/rollback_service.yaml",

inventory: "inventories/dev/hosts.ini",

credentialsId: <your git user credentials id>',

extras: '-e "date=' + "${env.DATE}" + ' env=' + "${params.environment}" + ' slack=' + "${env.SLACK_CALLBACK}" + ' dump_path=' + "/tmp" + '" -v',

tags: "run")

}

}

Summing up

Current solution is quite complex, but it allows you to run rollbacks both from Ansible via cli and from Jenkins. The second one is preferred, as you can see the user who approved the rollback.

Have a nice firefighting and hope you’ll never have a need in rollbacks!