[Distutils] A new, experimental packaging tool: distil

I've created a new tool called distil which I'm using to experiment with packaging functionality. Overview -------- It's based on distlib and has IMO some interesting features. With it, one can: * Install projects from PyPI and wheels (see PEP 427). Distil does not invoke setup.py, so projects that do significant computation in setup.py may not be installable by distil. However, a large number of projects on PyPI *can* be installed, and dependencies are detected, downloaded and installed. For those distributions that absolutely *have* to run setup.py, distil can create wheels using pip as a helper, and then install from those wheels. * Optionally upgrade installed distributions, whether installed by distil or installed by pip. * Uninstall distributions installed by distil or pip. * Build source distributions in .tar.gz, .tar.bz2, and .zip formats. * Build binary distributions in wheel format. These can be pure-Python, or have C libraries and extensions. Support for Cython and Fortran (using f2py) is possible, though currently distil cannot install Cython or Numpy directly because of how they use setup.py. * Run tests on built distributions. * Register projects on PyPI. * Upload distributions to PyPI. * Upload documentation to http://pythonhosted.org/. * Display dependencies of a distribution - either as a list of what would be downloaded (and a suggested download order), or in Graphviz format suitable for conversion to an image. Getting started is simple (documentation is at [2]): * Very simple deployment - just copy distil.py[1] to a location on your path, optionally naming it to distil on POSIX platforms. There's no need to install distlib - it's all included. * Uses either a system Python or one in a virtual environment, but by default installs to the user site rather than system Python library locations. * Offers tab-completion and abbreviation of commands and parameters on Bash-compatible shells. Logically, packaging activities can be divided into a number of categories or roles: * Archiver - builds source distributions from a source tree * Builder - builds binary distributions from source * Installer - installs source or binary distributions This version of distil incorporates (for convenience) all of the above roles. There is a school of thought which says that that these roles should be fulfilled by separate programs, and that's fine for production quality tools - it's just more convenient for now to have everything in one package for an experimental tool like distil. Actual Improvements ------------------- Despite the fact that distil is in an alpha stage of development and has received no real-world exposure like the existing go-to packaging tools, it does offer some improvements over them: * Dependency resolution can be performed without downloading any distributions. Unlike e.g. pip, you are told which additional dependencies will be downloaded and installed, before any download occurs. * Better information is stored for uninstallation. This allows better feedback to be given to users during uninstallation. * Dependency checking is done during uninstallation. Say you've installed a distribution A, which pulled in dependencies B and C. If you request an uninstallation of B (or C), distil will complain that you can't do this because A needs it. When you uninstall A, you are offered the option to uninstall B and C as well (assuming you didn't install something else that depends on B or C, after installing A). * By default, installation is to the user site and not to the system Python, so you shouldn't need to invoke sudo to install distributions for personal use which are not for specific projects/virtual environments. * There's no need to "install" distil - the exact same script will run with any system Python or any venv (subject to Python version constraints of 2.6, 2.7, 3.2 or greater). Bootstrapping pip ----------------- I've used distil to bootstrap pip, then used that pip to install other stuff. I created a fresh PEP 405 venv with nothing in it, used distil to install a wheel[3] for my distribute fork which runs on Python 2.x and 3.x, then used distil to install pip from PyPI. Finally, to test pip, I installed SQLAlchemy (using pip) from PyPI. See [4] for the transcript. I would welcome any feedback you could give regarding distil/distlib. There is of course a lot more testing to do, but I consider these initial findings to be promising, and worth sharing. If you find any problems, you can raise issues at [5]. Regards, Vinay Sajip [1] https://bitbucket.org/vinay.sajip/docs-distil/downloads/distil.py [2] https://pythonhosted.org/distil/ [3] https://bitbucket.org/vinay.sajip/distribute3/downloads/ [4] https://gist.github.com/vsajip/5243936 [5] https://bitbucket.org/vinay.sajip/distlib/issues/new