Ansible, Python 3, and distributions

This article brought to you by LWN subscribers Subscribers to LWN.net made this article — and everything that surrounds it — possible. If you appreciate our content, please buy a subscription and make the next set of articles possible.

The switch to Python 3 has been long—and is certainly far from over—but progress is being made in the form of Linux distributions that are switching their default. There are a number of high-profile projects with large, existing code bases (Mercurial and PyPy, for example) that have no imminent plans to switch or have not yet gotten around to it. Ansible is one of those projects, which makes it harder to support in distributions that have moved to Python 3 by default.

Fedora 23 is one such distribution. Problems with its Ansible support were raised by Dusty Mabe on the fedora-devel mailing list. He noted that a simple Ansible playbook required him to install four separate packages ( python for Python 2 and three others for the modules used by the playbook) in order to get it to run on a Fedora client (i.e. managed system). As he pointed out, the list of extra packages will vary for each user depending on the Ansible modules used in their playbooks, so there is no one-size-fits-all solution for the problem.

Further down the thread, Bill Nottingham gave a nice overview of how Ansible works and why it needs to lag in supporting Python 3.

Ansible works by connecting to other machines and sending across small bits of python to execute. In Ansible parlance, these are called 'modules'. What this means is that the version of python that you're using on the control machine needs to reasonably match the version of python on machines you're controlling, and that all bindings you use in your modules need to be of the same python version as the Ansible version you're using. If Ansible were to use python3, all module bindings would need to be python 3, and *all the managed machines would need to have python3 installed*.

Since Ansible needs to work with a lot of systems that only have Python 2 available (including RHEL 5 and derivatives that use Python 2.4), it needs to support the older Python versions. The bulk of the systems managed by Ansible are not on the more cutting edge community distributions, as Nottingham pointed out: "The percentage of people using Ansible to manage Fedora (and other python3-using-distros) doesn't justify moving Ansible to python3 at this time."

As far as the "pain" that Mabe reported, Adam Williamson suggested that Ansible playbooks could ensure the required dependencies were present on the remote system. It would be difficult or impossible for some kind of Ansible runtime meta-package to depend on all of the different Python 2 packages that might be needed, as was suggested by Orion Poplawski, since the list of possibilities is quite long—there is no list of "standard" modules, either.

For example, any Ansible client installation will need the python-dnf package in order to have Python 2 bindings for the DNF package manager to install packages. The rest of the system uses the python3-dnf package to do the same job, but for Python 3, of course. Other dependencies are essentially similar, though playbooks don't typically specify their Python dependencies.

While the PyPy and Mercurial packages (and others based on Python 2) can simply list the dependencies they need (including the python package for Python 2), Ansible has another dimension to deal with. It has an entire ecosystem of Ansible modules, which adds a layer of complexity.

Predictably, there were calls for Ansible to get "out of the stone age", as Kevin Kofler put it. He suggested a fork to make that happen, but it seems to be a fundamental misunderstanding of where Ansible is targeted—and used. There are really two Ansible installations to consider, as Florian Weimer pointed out:

I think there are two different levels of Python 3 support, one for the managing host (desirable but not essential), and one for the managed hosts (increasingly important if there are systems which only have Python 3 and cannot install Python 2).

The Ansible bug report for Python 3 support has a comment in the locked thread that mentions progress in the support for running the controlling (or managing) host using Python 3. In the comment, Toshio Kuratomi said that the Ansible core is moving toward a single code base that can support Python 2.6, 2.7, and 3.4. "We don't anticipate this to allow running ansible on python3 in the near future but we are not hostile to changes that help get us to this goal."

But support for RHEL 5 and its Python 2.4 interpreter is still a problem area. There are techniques to maintain code that needs to run with multiple Python versions, including 2.x and 3.x, but they do not support Python versions earlier than 2.6.

There are other distributions defaulting to Python 3 at this point, notably Arch Linux. The instructions for Ansible on Arch note the problem (and point to an Ansible FAQ entry), but do not directly address the module issue. Arch users have to be fairly savvy, though, so this may be less of a problem than it might be for some Fedorans—newbies in particular.

For Fedora 23, there isn't much the distribution can do. As Williamson said: "it's just...how ansible works?". Some information will be added to the release notes, pointing out the need for the python-dnf package for clients in order to be able to install anything using Ansible.

These kinds of hiccups are to be expected as distributions shift to Python 3. When the decision was made—many moons ago—to break backward compatibility for Python 3, it set all of this in motion. It has taken most of a decade to encounter the problems (Python 3.0 was released in December 2008), but they haven't actually been too daunting, at least so far. One suspects that other distributions will have an even easier ride as Arch and Fedora find (and fix) these rough spots.