[pypy-dev] PyPy support for Python 3 -- request for comments

Hello pypy-dev, in the past weeks, I and the other core developers have talked a lot about supporting Python 3 in PyPy. The task is huge and it's unlikely that it will be completed shortly based only on volunteer work, so we came up with the following proposal, which splits the work into several steps and sub-steps, togheter with an estimate of how much money is needed to complete each one. The plan is to publish the proposal on the blog, and publicly ask for donations, with the hope to collect enough to cover the whole work. However, before putting it on the blog we would like to ask for your comments/thoughts about it. Any feedback will be appreciated! Thank you, Antonio (on the behalf of the PyPy team) Python 3 on PyPy ================= The release of Python 3 has been a major undertaking for the Python community, both technically and socially. So far PyPy implements only the version 2 of the Python language, which creates a very dangerous potential for a community split. We believe that by supporting both versions of the language we will help to fill the gap. This project should help both the part of the Python community which is reluctant to use PyPy because it does not support Python 3, and the part which is reluctant to move to Python 3 because they are already PyPy users. However, porting PyPy to Python 3 requires a lot of work, and it will take a long time before we can complete it by relying only on volunteer work. Thus, we are asking the community to help with funding the necessary work, to make it happen faster. High level description ----------------------- The goal of this project is to write an interpreter that interprets version 3 of Python language. To be precise we would aim at having Python 3.2 interpreter together in the same codebase as python 2.7 one. At the end of the project, it will be possible to decide at translation time whether to build an interpreter which supports Python 2.7 or Python 3.2 and both versions will be nightly tested and available from nightly builds. The focus of this project is on compatibility, not performance. In particular, it might be possible that the resulting Python 3 interpreter will be slower than the Python 2 one. If needed, optimizing and making it more JIT friendly will be the scope of a separate project. Step 1: core language ---------------------- In this step, we implement all the changes to the core language, i.e. everything which is not in the extension modules. This includes, but it is not necessarily limited to the following items, which are split into two big areas: * **Sub-step 1.1**: string vs unicode and I/O: - adapt the existing testing infrastructure to support running Python 3 code - string vs bytes: the interpreter uses unicode strings everywhere. - the ``print`` function - ``open`` is now an alias for ``io.open``, removal of the old file type. - string formatting (for the part which is not already implemented in Python 2.7) - the _io module (for the part which is not already implemented in Python 2.7) - syntactic changes to make ``io.py`` importable (in particular: ``metaclass=...`` in class declarations) - **Estimate cost**: 35.000 $ * **Sub-step 1.2**: other syntactic changes, builtin types and functions, exceptions: - views and iterators instead of lists (e.g., ``dict.items()``, ``map``, ``range`` & co.) - new rules for ordering comparisons - removal of old-style classes - int/long unification - function annotations - smaller syntax changes, such as keyword-only arguments, ``nonlocal``, extended iterable unpacking, set literals, dict and set comprehension, etc. - changes to exceptions: ``__traceback__`` attribute, chained exceptions, ``del e`` at the end of the except block, etc. - changes to builtins: ``super``, ``input``, ``next()``, etc. - improved ``with`` statement - **Estimate cost**: 25.000 $ Note that the distinction between sub-steps 1.1 and 1.2 is blur, and it might be possible that during the development we will decide to move items between the two sub-steps, as needed. For more information, look at the various "What's new" documents: - http://docs.python.org/py3k/whatsnew/3.0.html - http://docs.python.org/py3k/whatsnew/3.1.html - http://docs.python.org/py3k/whatsnew/3.2.html **Total estimate cost**: 60.000 $ Step 2: extension modules -------------------------- In this step, we implement all the changes to the extension modules which are written in C in CPython. This includes, but it is not necessarily limited to: - ``collections``, ``gzip``, ``bz2``, ``decimal``, ``itertools``, ``re``, ``functools``, ``pickle``, ``_elementtree``, ``math``, etc. **Estimate cost**: this is hard to do at this point, we will be able to give a more precise estimate as soon as Step 1 is completed. As a reference, it should be possible to complete it with 35.000 $ Step 3: cpyext -------------- The ``cpyext`` module allows to load CPython C extensions in PyPy. Since the C API changed a lot between Python 2.7 and Python 3.2, ``cpyext`` will not work out of the box in the Python 3 PyPy interpreter. In this step, we will adapt it to work with Python 3 as well. Note that, even for Python 2, ``cpyext`` is still in a beta state. In particular, not all extension modules compile and load correctly. As a consequence, the same will be true for Python 3 as well. As a general rule, we expect that if a Python 2 module works with ``cpyext``, the corresponding Python 3 version will also work when this step is completed, although the details might vary depending on the exact C extension module. **Estimate cost**: 10.000 $