Debian plans for Python 3

This article brought to you by LWN subscribers Subscribers to LWN.net made this article — and everything that surrounds it — possible. If you appreciate our content, please buy a subscription and make the next set of articles possible.

Moving a distribution to Python 3 is a big job, but there is a clear deadline for Python 2: it will be unsupported by upstream in 2020 (or at least that is the plan). That may seem far away, but it is only roughly two Debian release cycles from now. By the time Debian 10.0 ("Buster") is released, all of Debian's tools, and most of its packages, should be using Python 3 if all goes according to plan. Paul Tagliamonte and a "shadowy Debian Python cabal" met at PyCon 2015 to discuss this issue with an eye toward creating a plan of attack.

The rudiments of that plan were outlined by Tagliamonte after the conference. The first step, he said, is to look at things that are completely under Debian's control, which is its infrastructure. He asked those who are working on infrastructure projects to send him a census of the tools, their dependencies, and their Python versions, so that he could start building a tree of dependencies.

He also suggested filing bugs for packages that support Python 3 upstream, but are not built that way for Debian, as well as for packages that do not have Python 3 support at all. Beyond that, he suggested porting any projects that only support Python 2 to support both with a hybrid 2/3 code base. Finally, he will be forming a "Python 3 porting team" to help move the code in Debian's infrastructure to Python 2/3.

Those are ambitious plans, of course, and, as evidenced by the first census response, there are still wrinkles to be worked out. Enrico Zini listed half a dozen services that he maintains (e.g. contributors.debian.org, debtags.debian.net) along with their dependencies, which were mostly things like the Django web framework, database access libraries, and other web-focused Python packages. He noted that he is in favor of switching to Python 3, in principle, but there are some barriers to doing so:

At the moment, I have a strong guarantee from the python core developers that my code WILL keep working until 2020, and I will NOT have that guarantee if I migrate to python 3. I would laugh at this, if it didn't make me cry first. See my comments at [4] So, if I am left to my own devices, I will *not* consider any porting effort until late 2019, to make sure that between here and 2020 I will need to do porting only once from 2 to 3, instead of who knows how many times from 2.x to 3.4, then from 3.x to 3.x+n at every new Debian stable release.

Zini is willing to work with the porting team, but he wants to be sure that the team will continue its effort past the initial port. If incompatible changes come in three or four years as part of Python 3.7, say, he wants someone to be responsible for whatever forward porting is needed. It is clear that Zini feels that he has been burned by Python's changes in the past and is happy to see the stability offered by the feature-frozen 2.7 branch.

Tagliamonte acknowledged Zini's concerns with regard to the porting team's help now and in the future. He suggested that the team could focus on making both the dependencies and Zini's services run on both 2 and 3:

You could continue the production deployment on Python 2, and also let some porting work go on to ease that transition later. I believe this would mostly take the form of a new python-six dependency, as well as evaluating [alternatives] to dead-end Python 2 only support libraries.

But Zini had some concerns about accepting patches that use the six compatibility library or, at least, that use it indiscriminately. He wants maintainable code moving forward:

I use dict.iteritems a lot, and I do not intend to see my code migrated to use six.iteritems because it makes code hard to read, and it then requires a [further] migration to make it readable again once there is a switch to python3. I can however take a migration to .items() for a temporary performance penalty that will go away without need of extra intervention after a switch to python3. So, patches that make code work in python3 without breaking idiomatic code, good. Patches that turn the code into an intermediate six-based language that looks neither like 2 nor 3, no thanks: that code will have to be maintained afterwards.

In addition, adding Python 3 support adds to the testing burden, Zini said. There were several suggestions of tools (e.g. tox, nose2) that would help, but the fact remains that someone has to run the tests and fix any fallout. He would like to see that be handled by the porting team (or someone else) at least until 2019 or so, when Python 3 is ascendant and Python 2 largely gone.

Beyond the porting question, Zini also noted that there might be some problems running the services in production atop Python 3. Debian infrastructure runs on Debian stable, so any packages required to run the services would need to be available for stable as well. As Scott Kitterman pointed out, the fact that there will be two releases of Debian in the interim makes the production transition easier. There is a goal to have all of the dependencies available in Debian 9.0 ("Stretch"), so that the final switch for 10.0 will go smoothly.

That discussion should provide a bit of the flavor of the work ahead for the Python 3 porting team. It is clear that the dependencies are the key piece of the puzzle at this point, and that is where the focus is. Tagliamonte has created a new py3porters-devel mailing list to discuss plans and coordinate action going forward. It is a big task, but one that Debian appears to be getting a good head start on.