[Python-Dev] Status of packaging in 3.3

Hi all, We need to make a decision about the packaging module in Python 3.3. Please read this message and breathe deeply before replying :) [Sorry this ends up being so long; Tarek, Georg, Guido, I hope you have the time to read it.] Let me first summarize the history of packaging in the standard library. (Please correct if I misremember something; this email expresses my opinion and I did not talk with Tarek or Alexis before writing it.) Two years ago the distutils2 (hereafter d2) project was started outside of the stdlib, to allow for fast-paced changes, releases and testing before merging it back. Changes in distutils were reverted to go back to misfeature-for-misfeature compatibility (but not bug-for-bug: bug fixes are allowed, unless we know for sure everyone is depending on them or working around them). Tarek’s first hope was to have something that could be included in 2.7 and 3.2, but these deadlines came too fast. At one point near the start of 2011 (didn’t find the email) there was a discussion with Martin about adding support for the stable ABI or parallel builds to distutils, in which Tarek and I opposed adding this new feature to distutils as per the freeze policy, and Martin declared he was not willing to work outside of the standard library. We (d2 developers and python-dev) then quickly agreed that distutils2 would be merged back after the release of 3.2, which was done. There was no PEP requested for this addition, maybe because this was not a fully new module but an improvement of an existing one with real-world-tested features, or maybe just because nobody thought about the process. In retrospect, a PEP would have really helped define the scope of the changes and the roadmap for packaging. Real life caused contributors to come and go, and the primary maintainer (Tarek at first, me since last December) to be at times very busy (like me these last three months), with the result that packaging is in my opinion just not ready. Many big and small things need more work: the three packaging PEPs implemented in d2 have small flaws or missing pieces (I’m not repeating the list here to avoid spawning subthreads) that need to be addressed, we’ve started to get feedback from users and developers only recently (pip and buildout devs since last PyCon for example) the public Python API of d2 is far from great, the implementation is of very unequal quality, important features have patches that are not fully ready (—and I do acknowledge that I am the blocker for reviews on many of them—), the compiler system has not been revised, tests are not all clear and robust, some of the bdist commands need to be removed, a new bdist format needs to be designed, etc. With beta coming, a way to deal with that unfortunate situation needs to be found. We could (a) grant an exception to packaging to allow changes after beta1; (b) keep packaging as it is now under a provisional status, with due warnings that many things are expected to change; (c) remove the unstable parts and deliver a subset that works (proposed by Tarek to the Pyramid author on distutils-sig); (d) not release packaging as part of Python 3.3 (I think that was also suggested on distutils-sig last month). I don’t think (a) would give us enough time; we really want a few months (and releases) to hash out the API (most notably with the pip and buildout developers) and clean the bdist situation. Likewise (c) would require developer (my) time that is currently in short supply. (b) also requires time and would make development harder, not to mention probable user pain. This leaves (d), after long reflection, as my preferred choice, even though I disliked the idea at first (and I fully expect Tarek to feel the same way). I’d like to stress that this is not as bad as it appears at first. We (I) will have to craft reassuring wording to explain why 3.3b1 does not include packaging any more, but I believe that it would be worse for our users (code-wise and PR-wise) to deliver a half-finished version in 3.3 rather than to retract it and wait for 3.4. And if we keep in mind that many people are still using a 2.x version, releasing in 3.3 or 3.4 makes no change for them: the standalone releases on PyPI will keep coming. Developer-wise, this would *not* mean that the considerable effort that went into porting and merging, and the really appreciated patches from developers such as Vinay, would have been in vain: even if packaging is removed from the main repo (or just from the release systems), there would be a clone to continue development, or the module would be added back right after the 3.3 release, or we would develop in the d2 repo and merge it back when it’s ready—this is really an implementation detail for the decision; my point is that the work will not be lost. Thanks for reading; please express your opinion (especially Tarek as d2 project lead, Georg as RM and Guido as BDFL).