Ansible Silo

If you expect reproducible outcome of an automation system, you not only need to make sure you have a specific version of the automation system itself, but also have fixed versions of all its dependencies. The most prominent Ansible dependency which can affect your plays would be Jinja2 but this applies to all involved components.

Silo not only removes moving parts by having 100% fixed dependencies hardcoded in a Docker image. It also enables you to switch Ansible to any version without affecting other users, therefore making it easy to test your playbooks and roles against new Ansible releases and run differing Ansible versions per playbook, project or user.

Furthermore you can bundle your playbooks (incl. configuration, roles, plugins etc) in a custom Docker image which inherits Silo and therefore generate a versioned, shippable, complete and self-contained executable package which runs your playbooks in any environment. (where you have access to a Docker daemon)

For convenience ansible-silo includes ansible-lint. Since ansible-lint uses the Ansible libraries it may react differently depending on the used Ansible version.

Demo

This recording is powered by asciinema player and demo-magic.

What problem does Silo solve?

If you expect reproducible outcome of an automation system, you not only need to make sure you have a specific version of the automation system itself, but also have fixed versions of all its dependencies. The most prominent Ansible dependency which can affect your plays would be Jinja2 but this applies to all involved components.

One approach to solve above problem are shared control hosts. Control hosts though add another problem: All teams and users who use the Ansible control hosts need to align on one specific Ansible version and it gets extremely complicated to update Ansible on the control hosts since all teams and users need to align and test all their roles and playbooks against the new version.

Silo not only removes moving parts by having 100% fixed dependencies hardcoded in a Docker image. It also enables you to switch Ansible to any version without affecting other users, therefore making it easy to test your playbooks and roles against new Ansible releases and run differing Ansible versions per playbook, project or user.

How it works

Silo bundles Ansible and all its dependencies in a Docker image.

To enable Ansible in the container to connect to remote hosts, your ~/.ssh folder is mounted and also, if available, your ssh auth socket (key forwarding) is mounted.

Starting the Silo container is a complex Docker command which needs to cover forwarding of environment variables and mounting required resources like the users ssh configuration. This complex command itself is included in the image and can be fetched by starting the container with the --run flag. The returned command then can be executed on the host which will start the container again with the correct parameters.

A bash script to easily trigger this process can automatically be installed by Silo when called with the --install flag.

The Silo container is not persistent, not running in the background. A new container is started for every Ansible call and is automatically removed after completion.

Standalone mode

Standalone mode means you run Silo as a replacement for Ansible. By default Ansible is installed in a Docker volume. The volume can be changed, so you can have multiple volumes with different Ansible versions, e.g. per user, per environment or per playbook.

Playbooks will be mounted from the local file system and are not part of Silo.

Bundle mode

A bundle is a Docker image which inherits the Silo Docker image.

In the bundle you can add your playbooks, roles, configuration and a specific Ansible version to the bundle.

A new bundle can easily be created by calling Silo with --bundle <bundle name> option.

Included software

Silo is based on Alpine Linux 3.6 and includes the following APK packages:

bash 4.3.48-r1

ca-certificates 20161130-r2

curl 7.55.0-r0

expat 2.2.0-r1

gdbm 1.12-r0

git 2.13.5-r0

gmp 6.1.2-r0

gosu 1.9-r0

libbz2 1.0.6-r5

libcurl 7.55.0-r0

libffi 3.2.1-r3

libxml2 2.9.4-r4

libxslt 1.1.29-r3

libssh2 1.8.0-r1

ncurses-libs 6.0-r8

ncurses-terminfo 6.0-r8

ncurses-terminfo-base 6.0-r8

musl 1.1.16-r13

musl-utils 1.1.16-r13

openssh 7.5_p1-r1

openssh-client 7.5_p1-r1

openssh-sftp-server 7.5_p1-r1

openssl 1.0.2k-r0

pcre 8.41-r0

perl 5.24.1-r2

py-netifaces 0.10.5-r3

py2-pip 9.0.1-r1

readline 6.3.008-r5

sqlite-libs 3.18.0-r0

sshpass 1.06-r0

sudo 1.8.19_p2-r0

yaml 0.1.7-r0

The following Python modules are installed via pip:

cffi 1.10.0

cryptography 2.0.2

ecdsa 0.13

enum34 1.1.6

httplib2 0.9.2

idna 2.5

ipaddress 1.0.18

jinja2 2.8

jmespath 0.9.3

markupsafe 0.23

netaddr 0.7.19

paramiko 1.16.0

pexpect 4.2.1

ptyprocess 0.5.2

pycparser 2.18

pycrypto 2.6.1

pyyaml 3.11

six 1.10.0

Installation

Prerequisites

You need to be on a system where you have installed Docker (minimum version 1.9).

Install ansible-silo

To install the ansible-silo executable along with Ansible replacements run:

docker run --interactive --tty --rm --volume " $HOME /bin:/silo_install_path" grpn/ansible-silo:2.2.0 --install

This command mounts your ~/bin directory so Silo can place its executables there. Select any location you like but make sure it is in your $PATH .

To install ansibe-silo for all users you can mount /usr/local/bin :

docker run --interactive --tty --rm --volume "/usr/local/bin:/silo_install_path" grpn/ansible-silo:2.2.0 --install

Uninstall ansible-silo

During installation two things happened:

A Docker image was downloaded

A file and a couple of symlinks were created

Where the files/links are stored depends on which path you have mounted during the installation process (see above, e.g. $HOME/bin or /usr/local/bin ).

You can use this command to delete all files and symlinks:

find -L " $( dirname $( command -v ansible-silo )) " -samefile " $( command -v ansible-silo ) " -exec rm -f {} +

All versions of the ansible-silo images can be deleted per:

docker rmi --force $( docker images -q grpn/ansible-silo | uniq )

Updating

It is important to understand, that by updating Silo you do not automatically switch the Ansible version. Ansible is stored in the Docker volume silo.$(whoami) . If you want to switch the Ansible version, you manually need to run the switch.

This also means you do not need to pull the latest version of the image to run a newer version of Ansible inside. You can run any Ansible version in any version of the Silo image.

To update the image run:

ansible-silo --update

This will pull the latest image from the Docker registry and automatically tries to replace the ansible* executables. If these are not writable by your user you can write them to a different location and later move then with sudo .

mkdir /tmp/ansible SILO_PATH = /tmp/ansible ansible-silo --update sudo mv /tmp/ansible/ * /usr/local/bin rm -rf /tmp/ansible

Silo will by default run the latest installed version of itself. You also can run any other version of Silo by simply passing in the version:

SILO_VERSION = 1.2.2 ansible-silo --version ansible-silo 1.2.2 ansible 2.3.0.0 ansible-lint 3.4.20 ansible installed on volume silo.some.user

Extending runner script

The Docker command which gets executed for calling Ansible is stored inside the image itself, so it cannot be modified. To inject additional parameters into the command you can define functions in ./.ansible-silo , your ~/.ansible-silo or globally in /etc/ansible/ansible-silo/ansible-silo file matching the pattern silo_* or _silo_* . The runner script will execute all silo_* and _silo_* functions and append their output to the Docker command.

For instance, if you need to mount an additional volume, you can add a method like this to your ~/.ansible-silo file:

silo_custom_volume_mounting () { local VOLUME_PATH = " $HOME /some/path" if [[ -n " $VOLUME_PATH " && -d " $VOLUME_PATH " ]] ; then echo "--volume ' $( cd " $VOLUME_PATH " && pwd -P ) :/tmp/custom-volume'" fi }

In Silo bundles you can add functions to the file bundle_extension.sh inside your bundle directory. To customize behavior per user or host, you also have the option to add functions to files matching the image name. If, for example, you run a bundle called foo-bar , Silo will search for the files ./.foo-bar , ~/.foo-bar and /etc/ansible/ansible-silo/foo-bar and append the output of all functions matching the pattern foo_bar_* to the Docker command.

Functions matching _silo_* will not be included in bundle mode. Functions matching silo_* will.

standalone bundle silo_* ✓ ✓ _silo_* ✓ ✗ image_name_* ✗ ✓

Installing custom software

You can install custom software in any Silo volume. The mountpoint for Silo volumes is /silo/userspace/ .

Inside any volume you will have a lib and a bin directory.

pip is pre-configured to install packages into the volume:

$ ansible-silo --shell pip install pbr == 3.1.1 /usr/lib/python2.7/site-packages/pip/commands/install.py:194: UserWarning: Disabling all use of wheels due to the use of --build-options / --global-options / --install-options . cmdoptions.check_install_build_global ( options ) Collecting pbr == 3.1.1 Downloading pbr-3.1.1.tar.gz ( 102kB ) 100% |████████████████████████████████| 112kB 7.9MB/s Installing collected packages: pbr Running setup.py install for pbr ... done Successfully installed pbr $ ansible-silo --shell which pbr /silo/userspace/bin/pbr

Configuration

Ansible Silo can be configured per bash environment variables. Variables will also be picked up from three files (bash):

./.ansible-silo (in current working directory)

(in current working directory) ~/.ansible-silo (in users home directory)

(in users home directory) /etc/ansible/ansible-silo/ansible-silo

For Silo bundles also the following files will be loaded:

./.bundle-name

~/.bundle-name

/etc/ansible/ansible-silo/bundle-name

This enables the user to define custom behavior per playbook location, per user and globally.

List of configuration options

SILO_DEBUG

If defined, enables debug mode. In debug mode Silo will list all SILO_* env vars and the Docker command which is executed to start the Silo container.

SILO_DEBUG = true ansible-silo --shell exit

Which will show something along these lines:

SILO vars: - SILO_DEBUG=true Runner file already exists. Executing: /tmp/ansible-silo-runner-2.2.0 "--shell" "exit" Executing: /usr/bin/docker run --interactive --tty --rm --volume "/home/daniel.schroeder/ansible-silo:/home/user/playbooks" --volume "silo.some.user:/silo/userspace" --env "SILO_VOLUME=silo.some.user" --hostname "silo.example.com" --volume /var/run/docker.sock:/var/run/docker.sock --privileged --volume "/home/some.user/.ssh:/home/user/._ssh" --volume "/tmp/ssh-6k3r1bCpCi":"/tmp/ssh-6k3r1bCpCi" --env SSH_AUTH_SOCK --env USER_NAME="some.user" --env USER_ID="1234" "grpn/ansible-silo:2.2.0" "--shell" "exit"

The first Executing line shows the location of the generated runner script. The last line shows the Docker command executed by the runner script.

SILO_DOCKER_CMD

The base Docker command that will be executed. This simply defaults to docker .

SILO_NO_PRIVILEGED

Disables privileged mode. Consequently will not forward the Docker socket into the container and any interaction with Docker will not work.

SILO_PATH

Can be used to specify a custom location where Silo starter scripts will be installed during update.

SILO_PATH = /tmp/ansible ansible-silo --update

SILO_VERSION

If set to a valid Silo version, that specific version of Silo container will be started. This does not change the installed ansible-silo command. Most functionality though is inside the container and not the starter script.

SILO_VOLUME

Specifies the name of the used Silo volume. This defaults to the name of the current user.

Usage

--version Show current Silo & Ansible version

$ ansible-silo --version ansible-silo 2.2.0 ansible 2.4.2.0 ansible-lint 3.4.20 ansible installed on volume silo.some.user

--switch Switch to any Ansible version

$ ansible-silo --switch v1.9.4-1 Switched to Ansible 1.9.4

The given version relates to any git tag or branch of the Ansible github repository. To switch to the development branch run:

$ ansible-silo --switch devel Switched to Ansible 2.4.0

--reset Resets a Silo volume

Will reset (delete) a Silo volume.

$ ansible-silo --reset

The volume can be specified by environment variable SILO_VOLUME:

$ SILO_VOLUME = "foo" ansible-silo --reset

--shell Log into container / execute command in container

You can log into the running Silo container by calling Silo with the --shell option. This can be used to install custom software in a Silo volume.

$ ansible-silo --shell [ ansible-silo 2.2.0|~/playbooks] $

All arguments after the --shell option will be directly executed.

$ ansible-silo --shell pip install pbr == 3.1.1 /usr/lib/python2.7/site-packages/pip/commands/install.py:194: UserWarning: Disabling all use of wheels due to the use of --build-options / --global-options / --install-options . cmdoptions.check_install_build_global ( options ) Collecting pbr == 3.1.1 Downloading pbr-3.1.1.tar.gz ( 102kB ) 100% |████████████████████████████████| 112kB 7.9MB/s Installing collected packages: pbr Running setup.py install for pbr ... done Successfully installed pbr

Run Silo with different Ansible versions

You can run multiple Ansible versions in parallel by installing Ansible in different volumes. By default, Silo will use the volume silo.<username> , e.g. silo.some.user .

The name of the volume can be changed by passing the environment variable SILO_VOLUME . The volume name will be prepended with silo. and automatically be created if it does not exist. It will contain Ansible 2.4.2.0, the latest version as of writing this document. To change the Ansible version in that volume run the switch command:

$ SILO_VOLUME = "1.9.6" ansible-silo --switch v1.9.6-1 Switched to Ansible 1.9.6 $ ansible-silo --version ansible-silo 2.2.0 ansible 2.4.2.0 ansible-lint 3.4.20 ansible installed on volume silo.1.9.6 $ SILO_VOLUME = "1.9.6" ansible-silo --version ansible-silo 2.2.0 ansible 1.9.6 ansible-lint 3.4.20 ansible installed on volume silo.1.9.6

Using Ansible

If you want to run playbooks or access any other resources like inventory files, make sure you’re currently located in the directory of those files. You cannot access files outside of your current working directory since only this directory will be mounted in the Silo container.

If you installed the Ansible scripts you can use Ansible the exact same way you usually would. Just call ansible , ansible-playbook , etc.

Examples

Run a ping on all hosts

ansible all -m ping # or ansible-silo ansible all -m ping

Run a playbook

ansible-playbook some-playbook.yml -i some-inventory # or ansible-silo ansible-playbook some-playbook.yml -i some-inventory

Show man page for the template module:

ansible-doc template or ansible-silo ansible-doc template

Run ansible-lint on a playbook:

ansible-lint some-playbook.yml # or ansible-silo ansible-lint some-playbook.yml

Disabling privileged mode

By default, Silo forwards the Docker socket into the container to be able to run Ansible against other containers. This requires the Silo container to run in privileged mode. To disable this you can define the environment variable SILO_NO_PRIVILEGED .

Bundle mode

Silo can also be used as foundation to package and distribute your playbooks as Docker images. You can create a new bundle by calling:

ansible-silo --bundle foo

This will create all required files for building a custom Docker image based on Silo inside the newly created folder foo .

Store your playbooks, roles, inventory, ansible.cfg etc. inside foo/playbooks and then call the build script to create the Docker image.

The foo package also inherits most of Silos functionality. To install an executable for the bundle run:

docker run --interactive --tty --rm --volume " $HOME /bin:/silo_install_path" foo:latest --install

Now you can simply call foo to run your playbooks.

All files inside foo can be modified by you. For instance you should define a specific Ansible version in the Dockerfile. Have a look at the generated README.md inside your package for detailed description of the contained files.

FAQ

Why do I always have to enter my SSH key passphrase when Silo starts?

On OS X, forwarding of the SSH authentication socket currently is not possible. Therefore Silo cannot use your ssh agent, even though it is forwarded to the container. If you have a password protected SSH key, you need to enter it once after the container is started. Since Silo is not persistent you have to enter it on every Silo run.

Troubleshooting

If anything goes wrong, try to reset your Silo volume.

ansible-silo --reset

You can see the actual generated and executed docker run commands and all defined SILO_* environment vars by enabling debug mode.

Versioning

Ansible Silo uses SemVer. Since Ansible Silo is the promise of a stable environment, we consider the smallest update of any contained dependency to be a potential breaking change and indicate such change by incrementing the major version number.

Version history

v2.2.0 (Jun 8, 2018)

Adds support for non-tty environments.

Adds ansible-inventory command.

command. Adds support for ssh socket forwarded via uber/ssh-agent-forward (pinata).

v2.1.1 (Jan 15, 2018)

Fixes name of example function in bundle extension.

v2.1.0 (Jan 9, 2018)

Adds support for bundle extensions.

v2.0.4 (Jan 8, 2018)

Updates default Ansible version to 2.4.2.0.

Consequently updates ansible-lint to 3.4.20 as 3.4.13 is incompatible.

to 3.4.20 as 3.4.13 is incompatible. Moves ansible-lint into userspace, so the version in future can/must be managed by the user through pip .

. Adds new ansible-config command.

command. Adds support for environment variable ANSIBLE_VAULT_PASSWORD_FILE .

v2.0.3 (Sep 20, 2017)

Adds Python module ncclient 0.5.3

Updates default Ansible version to 2.4.0.0.

v2.0.2 (Sep 8, 2017)

Switches git cloning of Ansible repository from git: protocol to https:

v2.0.1 (Sep 6, 2017)

Adds jmespath and therefore fixes support for JSON Query Filter

v2.0.0 (Aug 31, 2017)

Updates Alpine Linux from 3.4 to 3.6 - as well uses latest apk’s.

Adds support for pip module installation.

Introduces userspace in volumes. Now custom software can be installed beside Ansible.

Loads .ansible-silo (or .bundle-name ) file from current working directory.

(or ) file from current working directory. Adds Docker support (Docker in Docker / DinD) - now can manage Docker containers and run Ansible against containers.

Silo container by default now is started in privileged mode.

Adds sshpass to support Ansible authentication via password.

to support Ansible authentication via password. Introduces new --reset option. Can be used instead of docker volume rm .

option. Can be used instead of . Improves output in installation routine - now shows image name and version from which software is installed.

Updates default Ansible version to 2.3.2.0.

Changes runner script file name - now is based on image-name and -version. This permits to skip runner script creation if it already exists, therefore improves container start time.

Prevents forwarding of Silo exit codes as env vars into the container.

Prevents forwarding of GIT_* env vars into the container to prevent potential git conflicts.

env vars into the container to prevent potential git conflicts. Debug output now lists all SILO_* env vars.

env vars. Fixes exit code declaration.

Fixes order in PYTHONPATH .

. Removes ansible-lint from bundle --version output.

from bundle output. Adds pyyaml .

v1.3.3 (July 20, 2017)

Converts starter scripts to symlinks

Fixes help message command name

v1.3.2 (July 17, 2017)

Fixes version number in automated Docker build

v1.3.1 (July 13, 2017)

Initial public release

Development

Build pipeline

The ansible-silo image is an automated Docker build triggered by Travis CI, whenever a tag passed all tests. This means, to release a new version only a new tag in the form v1.2.3 needs to be released.

There are custom build hooks in ./hooks which will be triggered by the automated Docker build process.

The ansible-silo-base image needs to be built and uploaded manually. You can do this by running a command like:

make ansible-silo-base push-base

The version of the base image is hardcoded in the Makefile. Update accordingly if you plan to release a new base image.

For testing purpose you can also manually build the ansible-silo image by running a command like:

make ansible-silo

Testing

Functional tests are implemented through bats. (0.4.0) After installing bats call:

make test-function

Be aware, tests modify the Ansible version of your default ansible-silo volume!

Code style tests are implemented via shellcheck (0.3.5). After installing shellcheck call:

make test-style

We also validate URLs inside all files for positive results via awesome_bot (1.17.1). After installing awesome_bot call:

make test-links

To run all tests call:

make test

Base image

The APK package repository of Alpine only holds the very latest version of a package. This makes it currently impossible to install exact package versions and building of the image would fail once a new version of a package was released.

To ensure we are never forced to update any dependency when we build the silo Docker image, all APK dependencies are stored in the Docker image ansible-silo-base. If required, this image can be built and uploaded with the command make ansible-silo-base push-base . Make sure to afterwards update the tag in the Dockerfile and release a new version of ansible-silo.

License