This edition contains the following feature content:

This week's edition also includes these inner pages:

Brief items: Brief news items from throughout the community.

Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

A short sub-thread on the python-ideas mailing list provides some "food for thought" about the purpose and scope of that list, but also some things to perhaps be considered more widely. When discussing new features and ideas, it is common for the conversation to be somewhat hypothetical, but honing in on something that could be implemented takes a fair amount of work for those participating. If the feature is proposed and championed by someone who has no intention of actually implementing it, should the thread come with some kind of warning?

The thread in question started in mid-June with a query from Thomas Güttler about why the socket module returns plain tuples rather than named tuples. The reception to the idea was mostly positive and there were some discussions of how it might be done; Guido van Rossum indicated that he would be favorable to the change as well. But, apparently Güttler was not actually planning to implement the change, as he currently does not have the time to do so.

That led core developer Brett Cannon to ask that participants be upfront about their interest and ability in implementing a feature when posting about it to python-ideas. If someone is just posting a hypothetical that they have no intention of implementing, that should be clear from the outset:

It's implicit that if you have an idea to discuss here that you are serious enough about it to see it happen, so if that's not the case then do say so in your first email (obviously if your circumstances change during the discussion then that's understandable). Otherwise people will spend what little spare time they have helping you think through your idea, and then find out that the discussion will more than likely end up leading to no change because the most motivated person behind the discussion isn't motivated enough to actually enact the change.

It is also important to note if the person posting does not have the requisite skills to implement their idea, he said. In the past, he said, they have gotten all the way to the Python Enhancement Proposal (PEP) stage before realizing that the person pushing the change did not know C, which was critical to implementing it. So the PEP (and the work that went into it) was effectively wasted effort. He acknowledged that folks might not know the requirements for implementing their idea right away, but felt that once that became clear, it was important to pass that information along:

Otherwise I personally feel like I was tricked into a discussion under false pretenses that the OP was motivated enough to put the effort in to see their idea come to be. Had I known to begin with that no one was actually stepping forward to make this change happen I would have skipped the thread and spent the time I put in following the discussion into something more productive like reviewing a pull request.

His words may sound a bit harsh—he did admit to a certain amount of frustration in another post—but the overarching issue is salient. Everyone who participates in a project has to try to intelligently allocate their time; it is not surprising that some might want to spend it on features and ideas that have some likelihood of seeing the light of day. While Brendan Barnwell said that he understood Cannon's points, he worried that most or all core developers (and others who might be able to implement feature ideas) might end up ignoring ideas from those who can't implement them:

You seem to be saying that (some of) those who do know how to implement stuff would like to be able to ignore discussion from anyone who doesn't know how to implement stuff. That's certainly anyone's prerogative, but I think it would be a shame if this resulted in a bifurcation of the list in which ideas can't reach the attention of people who could implement them unless they're proposed by someone who could do so themselves.

Based on the description of python-ideas, which specifically mentions moving more "baked" proposals to python-dev, Barnwell thought that some of the usefulness of the list might be lost if most developers were to ignore ideas from non-implementers. Beyond that, some of the discussions on the list do not really concern implementation details:

My impression of discussion on this list is that a good deal of it doesn't really have to do with implementation at all. It has to do with the proposal itself in terms of how it would feel to use it, hashing out what its semantics would be, what the benefits would be for code readability, what confusion it might create etc. --- in short, discussion from the perspective of people who USE Python, not people who implement Python. I think that's good discussion to have even if the proposal eventually stalls because no one with the right skills has the time or inclination to implement it.

Van Rossum more or less agreed with Cannon in a one-line response: "IMO we shouldn't discuss features without giving thought to their implementation." Paul Moore concurred as well.

The risk that Barnwell describes is real, though. Python-ideas has long been a freewheeling discussion forum for all kinds of thoughts about Python, some sensible, some out there a ways but plausible, and some that are completely off the wall (or are treading on well-settled decisions about the language, meaningful indentation, for example). It is also the recommended place to post feature requests. If anyone capable of actually implementing changes tunes out everything that comes from the "peanut gallery", valuable ideas, thoughts, and features may well be lost.

It is a difficult balance to strike; developers need to prioritize their time and attention, for sure, but hopefully some or many will find a bit of time to look at threads that come from Python users, rather than implementers, as Barnwell put it. Even though Cannon said he will be ignoring threads that are not on an "implementation track", one might guess that he will still occasionally look in on ideas in his areas of interest—no matter who proposes them. One would also guess there are examples of ideas that were initially described by someone who could never make them happen but that were picked up and run with by someone who could. That is part of the advantage to doing development in the open: it is open to all comers. It is totally understandable that hypothetical ideas can be draining at times, but sometimes they can lead somewhere interesting. Finding the balance between the two is important—hopefully participants on python-ideas are able to do so.

Comments (29 posted)

Software patents may not have brought about the free-software apocalypse that some have feared over the years, but they remain a minefield for the software industry as a whole. A small-scale example of this can be seen in the recent decision by the Apache Software Foundation (ASF) to move a license with patent-related terms to its "Category-X" list of licenses that cannot be used by ASF projects. A number of projects will be scrambling to replace software dependencies on a short timeline, all because Facebook wanted to clarify its patent-licensing terms.

Patent grants

Facebook has open-sourced a large number of projects that it uses internally; the company's preferred license is the permissive three-clause BSD license. This license says nothing about patents, but it has often been argued that any entity releasing software under the BSD license is making an implicit grant for any patents required to use that software. Otherwise, the permission for "redistribution and use in source and binary forms" would lack meaning. Other licenses, including GPLv2, are also generally assumed to contain an implicit patent grant, but this is an idea that lacks testing in court.

The problem with untested legal hypotheses is that they are about as reliable as untested software. While the implicit grant remains unproven, one cannot discount the prospect of an evil entity releasing software under a free copyright license with the intent of attacking the users of that software for patent infringement. Companies that worry about such things are thus much happier with an explicit patent grant. The implicit patent grant idea also can rub patent holders the wrong way; they would prefer to retain more control and, if they must grant patent-use rights when they release software, they would like to do so in a way that is revocable should (from their point of view) the need arise.

That is the reasoning that leads to Facebook's "additional grant of patent rights" that it attaches to its software releases. It explicitly grants a right to any Facebook patents that might otherwise be infringed by the software. But it also contains a patent retaliation term, in that anybody who asserts a patent against Facebook (for any reason) or against other users of the software (for patents allegedly infringed by the software itself) can have their patent license terminated. "Assertion" is the word used in the grant; things do not have to get to a point where a lawsuit is filed before the termination can happen.

The Apache Software License 2.0 (ASL) contains similar patent-grant language, but termination only happens when actual litigation is initiated. It is also, crucially, more focused: the retaliation clause is restricted to patents (alleged to be) infringed by the licensed software itself. The Facebook grant, instead, terminates on any patent assertion against Facebook, even if it has nothing to do with the software covered by the grant. This language feeds fears that, if a company uses Facebook code in its products, it must grant Facebook access to all of its patents or risk license termination. That, too, is an untested legal hypothesis and there does not appear to be any evidence that Facebook wants to use the grant in that manner, but it still raises eyebrows.

Category-X

In April, Jeff Jirsa raised an issue with the Apache legal community. The Apache Cassandra project was considering adding Facebook's RocksDB key-value database as a storage engine, but was wondering about the patent grant. In June, the community started discussing the issue and didn't much like what it saw. The asymmetric nature of the patent grant, in particular, was deemed to be incompatible with the ASL.

On July 15, ASF director and vice president of legal affairs Chris Mattmann issued a decree stating that the Facebook BSD+patents license had been placed into the dreaded Category X and that, as a result, Apache projects could not incorporate or depend on software that uses that license. For any projects that had already released code with such a dependency (and there were a few in the case of RocksDB), a temporary reprieve — until August 31 — was granted. In the meantime, such projects need to add a notice to their repositories that the Facebook patent grant is present there.

That would seem to be the end of the line for the use of RocksDB in Apache projects — except that RocksDB changed its license in response to the Category-X designation. It is now dual-licensed under the ASL and GPLv2. Once an official release has been made under that license, RocksDB will once again be usable by Apache projects.

The story doesn't quite end there, though: perhaps a bigger issue is React, which is also under the Facebook license regime. React is used by Apache projects like CouchDB; moving away from it will not be an easy thing for the CouchDB developers to do in the next month. It is thus unsurprising that the CouchDB developers have asked the React project to consider a license change similar to what RocksDB has done. That change is under discussion as of this writing; it's not clear that the odds of it happening are particularly high, though, so the CouchDB developers are looking at alternatives.

This sort of licensing disagreement is unfortunate; it divides our community into separate groups that are unable to share their software. Since sharing the software is one of the major points behind this whole exercise, that could be seen as a significant bug in how we operate. Perhaps it is unsurprising that an increasingly large part of the community seems to want to just toss the software out there and do away with licensing altogether. Outside of copyleft concerns, though, most of these disagreements are really artifacts of the legal regime in which the community operates. That regime changes slowly at best, so we can expect to be dealing with issues like this one for some time yet.

Comments (32 posted)

It has been nearly one month since the Stack Clash vulnerability was disclosed and some hardening measures were rushed into the 4.12 kernel release. Since then, a fair amount of work has gone into fixing problems caused by those measures and porting the result back to stable kernel releases. Now, it seems, the kernel developers are considering taking a different approach entirely.

Stack Clash is generally seen as a user-space problem: a combination of large on-stack allocations and the lack of stack probing open up opportunities for an attacker to jump over the guard page at the end of the stack. Fixing those problems (and deploying the fixes widely) will take some time; meanwhile, it was thought, systems can be protected by replacing the kernel-provided guard page with a 1MB (or larger) guard region that, hopefully, cannot be jumped over.

That guard region should be invisible to most programs, but it has created a surprising number of problems for some applications. A number of those issues have been worked around, but one has proved difficult to fix; unfortunately, that one is LibreOffice, which can crash on 32-bit systems when running Java components. The issue is a bit complicated but, in short, Java wants to enable code execution in memory located immediately below the stack, which runs afoul of the guard region. This, as Linus Torvalds noted, is a problem:

We really can't be breaking libreoffice. That's like a big classic no-no - it affects normal users that simply cannot be expected to work around it. For them, it's a "my office application no longer works" situation, and they just think the system is flaky.

There is also the lingering fear that other, not-yet-discovered regressions lurk in user-space applications, regressions that might not surface until somebody does a kernel upgrade months or years from now. So, perhaps, adding a large guard region to work around a user-space problem is not the best answer.

On most systems, the resource limit (rlimit) mechanism restricts the stack to a maximum of 8MB of memory. The hard limit tends to be much larger, though, so any unprivileged process can raise the effective stack limit to a larger value — as high as "infinite". The posted exploits for Stack Clash work by raising the stack rlimit, then running a setuid program with a vast array of arguments to fill that huge stack. If that program can be prodded into overflowing the stack without hitting the guard page, it may be possible to overwrite heap data beyond the stack and, from there, take over the program and make use of its privileges.

There may be other ways to get a setuid program to exhaust its stack, but the command-line arguments method is easy and readily under the control of an attacker. (Note: the overflow is relatively easy; a successful exploit is harder). That suggests that the bulk of the Stack Clash exploits can be headed off by preventing the execution of setuid programs with a stack that's both (1) close to the heap area and (2) nearly full at the outset.

Kees Cook took a stab at that problem with this patch attacking the first of those two points. The idea was that, when a process is about to run a setuid program, the stack limit should be reset to a reasonable value; the value Cook chose was whatever the init process has. This patch would not prevent stack exhaustion (indeed, it might cause it if there are setuid programs needing a huge stack), but it would keep the stack from growing large enough to impinge on a heap area.

That patch didn't get far, though, since Torvalds disliked it. One of his complaints was that special-casing setuid programs would be likely to lead to bugs or inadequate protection, since the relevant code would see relatively little use. So Cook's next attempt took a different tack: it places an upper bound on the amount of stack memory that can be occupied by a program's arguments at exec() time. In particular, that limit is 75% of the default stack limit, or 6MB, regardless of the current stack limit. This patch has been merged for 4.13; it's not clear whether this change will find its way into the stable updates to earlier kernel releases.

Limiting stack use by arguments should suffice to block a lot of attacks but, as it turns out, there's still a desire to enforce a limit on the size of the stack for setuid programs. One reason for that might be fears of some sort of pathological behavior that could be exploited to force a setuid program to overflow even a huge and mostly empty stack. But it also turns out that, if the stack rlimit is set to "infinity", the kernel will change the layout of a program's memory areas. A large stack limit suggests that the stack itself is likely to be large, so the kernel maps other memory areas low in the address space to preserve room for the stack to grow into. If, instead, the stack is limited, the kernel will map those areas at higher locations. As a result, the stack rlimit gives an attacker a bit of control over how the target program's memory is laid out — not a desirable thing to do.

Thus this patch series, which applies a maximum 8MB stack limit on setuid programs. These patches, posted on July 10, have not yet been merged; applying this limit required a surprising number of changes to the core exec*() system-call code, so more than the usual amount of review is indicated. There would appear to be general agreement on the goal, though, so this change seems likely to find its way into the mainline eventually. There has been some talk of allowing a larger stack via an annotation in the binary file, but that has not been implemented and may not be without a demonstrated need.

At this point, nobody has said whether these changes will be enough to allow the removal of the larger guard region from the stack. Returning to the previous layout semantics would ease a lot of worries about regressions that might turn up months or years in the future, though, so it's not hard to see why the idea has appeal. It would seem that at least some of the kernel's internal memory-layout policies have become a part of the user-space ABI, so they need to be preserved if possible.

Comments (18 posted)

By the end of the 4.13 merge window, 11,258 non-merge changesets had been pulled into the mainline repository — about 3,600 since the first half of this series was written. That is nowhere near the 12,920 changesets that landed during the 4.12 merge window, but it still makes for a typically busy development cycle. What follows is a summary of the more interesting changes found in those last 3,600+ changesets.

Some of the more significant user-visible changes are:

The mechanism for better reporting of writeback errors described in this article has been merged. This should result in more reliable reporting of errors to user-space applications. (This is in addition to the error-code refactoring changes reported last week.)

The ext4 filesystem has a new "largedir" option that increases the number of files that can exist in a single directory. Users who find the current limit of about 10 million files a bit constraining can enable this feature and raise the limit to around 2 billion. That should hold most people for the next few years.

The storage of extended attributes in ext4 filesystems can be optionally changed to allow many more attributes to be stored per file, and those attributes can each be up to 64KB in length. There is also a new deduplication feature that allows the filesystem to store a single copy of attributes that are applied to multiple files.

The f2fs filesystem now supports disk quotas.

The overlayfs union filesystem has a new "index directory" feature that makes copy-up operations work without breaking hard links.

NFS filesystems can now be re-exported over NFS. [ Note : that turns out not to be entirely true. Open-by-handle support has been added, but full re-exporting is still not possible.]

: that turns out not to be entirely true. Open-by-handle support has been added, but full re-exporting is still not possible.] New hardware support includes: Clock : TI system control interface clocks, Allwinner DE2 and A83T clock-control units, Qualcomm IPQ8074 global clock controllers, Broadcom Stingray clocks, and Cortina Systems Gemini clocks. Graphics : STMicroelectronics display controllers, PrimeCell PL111 LCD controllers, Allwinner display engine 2.0 mixers, Allwinner A10 HDMI controllers, and Innolux P079ZCA panels. PCI : HiSilicon Kirin series PCIe controllers, MediaTek PCIe controllers, and Sigma Designs Tango SMP8759 PCIe controllers. Miscellaneous : EXAR XRA1403 16-bit GPIO expanders, TI LP87565 GPIO controllers, TI LP87565 power-management ICs, Intel Cherry Trail Whiskey Cove power-management ICs, Broadcom SBA RAID engines, Faraday Technology FTDMAC020 DMA engines, D-Link DIR-685 touchkeys, STMicroelectronics STMFTS touchscreens, Renesas RZ/A WDT watchdog timers, STMicroelectronics STM32 independent watchdog timers, Socionext UniPhier watchdog timers, Aspeed I2C controllers, ZTE ZX2967 I2C controllers, IDT PCIe-switch non-transparent bridges, Microchip 23K256 SRAM chips, and Broadcom STB wake timers.



Changes visible to kernel developers include:

The kernel self-testing framework has begun a transition to the TAP13 reporting protocol. Not all tests have been converted, but the process is well underway.

The kernel build process has been changed to use "thin archives", which are described in the ar manual as: gnu ar can optionally create a thin archive, which contains a symbol index and references to the original copies of the member files of the archive. This is useful for building libraries for use within a local build tree, where the relocatable objects are expected to remain available, and copying the contents of each object would only waste time and space. The result of this change should be slightly faster and more space-efficient kernel builds.

manual as: The KASAN tool now works with hotplugged memory.

The __GFP_REPEAT memory-allocation flag, which has been the subject of discussion for years, has finally been removed. In its place is __GFP_RETRY_MAYFAIL which, it is hoped, better describes how it works. See this commit for a discussion of the options that exist for telling the allocator how hard it should try to satisfy any given request.

memory-allocation flag, which has been the subject of discussion for years, has finally been removed. In its place is which, it is hoped, better describes how it works. See this commit for a discussion of the options that exist for telling the allocator how hard it should try to satisfy any given request. The wait_for_random_bytes() API, described in this article, has been merged.

Note that anybody waiting for RDMA changes in 4.13 will be disappointed; that subsystem missed the merge window this time around and will have to wait for 4.14.

The 4.13 kernel is now in the stabilization cycle, with the final release mostly likely to happen in early September.

Comments (10 posted)

An under-the-radar proposal to stop building i686 kernels for Fedora led to a discussion about dropping support for 32-bit x86 hardware. Any of the hardware that needs these kernels is quite old, but participants in a thread on the Fedora devel mailing list noted that those systems still exist—some run Fedora. As the discussion progressed, though, it became clear that the Fedora i686 kernel has been in rough shape for some time now.

Florian Weimer raised the issue on July 11, likely inadvertently. He pointed to a proposal to stop building i686 kernels for Fedora 27 and noted that if that were to be adopted, it would mean that all x86 hardware still supported by Fedora would have the SSE2 instructions. That would allow the compile flags to be updated to add SSE2 before the "mass rebuild" for that release.

The "drop i686" proposal was marked as "not a system-wide change", but several in the thread objected to that designation. For example, Chris Adams pointed out that "it would remove a supported architecture completely". In addition, several pointed to Atom-based systems that are 32-bit, some of which are still being sold (mostly in the embedded space). Those CPUs do have SSE2, so it is possible that Weimer's request to enable SSE2 support would still make sense. Most other hardware that would be unable to run Fedora without an i686 kernel is at least ten years old.

But the Fedora i686 kernel has been in a state of benign neglect for several years. Josh Boyer said that the kernel team has not been doing much, if anything, for i686 kernels lately:

Anyone with 32-bit hardware is going to be against this change. It is a known downside. It also doesn't change the fact that i686 kernels are in a zombie state, where the kernel team does not actively support them and the community has not significantly stepped up to do so. That approach was done quite a while ago, and explicitly communicated.

Boyer concluded by saying that it was "basically luck" if i686 kernels are still working. In fact, Richard W. M. Jones said that he had stopped testing libguestfs with i686 kernels because they were broken most of the time when he tried them using QEMU; nobody was interested in fixing the bugs, he said. Laura Abbott added some perspective on the kinds of bugs that have appeared in i686 kernels; she said that the best way forward was to work with the upstream kernel community:

My overall concern with continuing to ship i686 kernels is any participants need to be active in advocating upstream. The only reason certain drivers or hardware are supported upstream is because of one or two people who stubbornly refuse to let it go. i686 is nowhere close to actually being deleted but fixing reported bugs in bugzilla is not a sustainable solution without at least some participation upstream to keep 32-bit healthy as Fedora relies very very heavily on upstream.

Ralf Corsepius posited that the lack of interest in i686 for Fedora really stemmed from a lack of interest by Red Hat. Fedora project leader Matthew Miller did not really disagree with that, saying that it is easier to support architectures that a company is paying someone to care about. Since Red Hat is not doing so, and no other company has stepped up, i686 kernels have languished. On the flipside, though:

I guarantee you that if some non-Red Hat person showed up and said "Hey, I'm here to work on i686 N hours per week", we would say "awesome", not "Red Hat doesn't care".

It is not just the kernel that is suffering, either. Adam Williamson pointed out that there is a ten-month-old bug for an installer feature that does not work correctly on 32-bit x86 kernels. Part of the problem is that there is nowhere for him to refer the bug: "There is no i686 WG/SIG, no more informal but well-known 'group of people who care about and help fix i686 bugs', so I can't do it for that bug", he said in another post.

Hans de Goede seemed to be taken aback by the "zombie" status of the i686 kernel. Boyer pointed to a post he made in February 2015 that laid out the problem from the kernel team's perspective and asked for the community to step up to help fix i686 bugs. Others in the thread listed additional places where the topic had come up as well. No one stepped up, which meant that i686 bugs simply languished.

But De Goede thought that Boyer's message was too much of a "generic, non specific call for help". What would be more helpful, he said, is "a concrete list of things people who care about i686 can work [on]". While Boyer agreed that kind of a list would be useful, he asked: "Are you volunteering to triage all the bugs and create it and maintain that list?" De Goede didn't directly address that question, but did recognize that an i686 SIG needed to be formed in order to move forward. He asked for some guidelines on what needed to happen in order to keep i686 kernels alive for Fedora 28 (the proposal to drop i686 kernels was submitted too late to be considered for Fedora 27).

Miller promptly provided that guidance, at least in draft form; the Fedora Engineering Steering Committee (FESCo) will presumably finalize the guidelines at an upcoming meeting. The idea is that if there is no functional i686 SIG by the Fedora 27 release, the kernel will be dropped for Fedora 28. The criteria for what "functional" means will need to be hammered out.

Corsepius complained that getting rid of i686 kernels showed that Fedora is not really a community-driven distribution. But Boyer strongly disagreed with that:

Community driven doesn't mean "I complain and someone else does the work so I get what I want." If you would like to see i686 kernels continue to be supported, please step up and volunteer to lead that effort. We asked for community help several releases ago and got none.

Moving forward, Stephen J. Smoogen has put out a call to gauge interest in forming an x86_32 SIG. He created a wiki page and listed some preliminary steps that would need to be taken. There has been a bit of activity on the wiki and some interest shown in the thread, but nothing overwhelming at this point. There was interest in a mailing list, so Smoogen created x86@lists.fedoraproject.org on July 18; the archives are rather sparse, so far, but it has only been a day or so.

It is unfortunate when older hardware gets left behind though, as Andy Lutomirski mentioned in the thread, the upstream kernel has good support for i686 (and even i486) these days. But distributions have to apportion their efforts in ways that make the most sense for them. Quite a few Linux distributions are eliminating 32-bit builds, so Fedora is hardly alone if it takes that route. But "community driven" also means that some parts have to be community supported, too, even for distributions like Fedora that have a large company behind them.

Comments (18 posted)

CPython is the reference implementation of Python, so it is, unsurprisingly, the target for various language-extension modules. But the API and ABI it provides to those extensions ends up limiting what alternative Python implementations—and even CPython itself—can do, since those interfaces must continue to be supported. Beyond that, though, the interfaces are not clearly delineated, so changes can unexpectedly affect extensions that have come to depend on them. A recent thread on the python-ideas mailing list looks at how to clean that situation up.

On July 11, Victor Stinner floated a draft of an as yet unnumbered Python Enhancement Proposal (PEP) entitled "Hide implementation details in the C API". The idea is to remove CPython implementation choices from the API so that different experimental choices can be made while still supporting the C-based extensions (NumPy and SciPy in particular). As he noted, other attempts to provide an alternate Python implementation (e.g. PyPy), which are typically created to enhance the language's performance, have largely run aground because they cannot directly support these all-important extensions.

In the draft, he mentioned a few possible options that could be tried if the C API was modified, including switching to indirect reference counts, removing the global interpreter lock (GIL), or changing the garbage-collection scheme. He described some of the history of Python forks and alternate implementations, many of which were blocked by the C API exposing too much of CPython's internals. The pre-PEP then went on to list some concrete steps toward splitting and rationalizing the Python C API.

To start, the Include/ directory for CPython would be split into three, one for each API: python for the existing C API, core for the internal API for CPython, and stable for the existing stable ABI (that extensions can rely on staying unchanged, though it is not really used, according to Stinner). Next up, the packaging tools would get an option for extensions to choose the API to use when they are built.

The final three steps would slowly move implementation details out of python , while still ensuring that extensions will build and function. That will require something of an iterative approach: alternately removing things from python and fixing the extensions. Eventually, the new restricted python API would be the default for all extensions. He also included an alternate path: leave the existing core API as the default, but provide an alternate API as an option at build time. That would mean that two Python binaries would be distributed for each release, one using the compatible API and another that would be faster but not compatible with all existing extensions.

Stinner prefaced his draft with some performance-related justifications, including a link to coverage of his 2017 Python Language Summit session. He is concerned about Python's performance and believes that the C API blocks various optimizations that might be applied to speed it up. He said:

This is the first draft of a big (?) project to prepare CPython to be able to "modernize" its implementation. Proposed changes should allow to make CPython more efficient in the future. The optimizations [themselves] are out of the scope of the PEP, but some examples are listed to explain why these changes are needed.

Nick Coghlan took issue with the use of "needed" with regard to performance improvements. He suggested that the status quo is a result of people not recognizing one of the best ways to increase the performance of a Python application: rewriting the performance-critical pieces in another language. "[...] So folks mistakenly think they need to rewrite their whole application in something else, rather than just selectively replacing key pieces of it." He pointed to Cython (which is used in parts of SciPy and elsewhere) as a known way to get C-level performance from Python. So there are differences of opinion about how necessary these potential performance enhancements are, he said.

However the reorganization of the API to more clearly specify what is (and is not) an external interface is "an admirable goal", Coghlan said, which will allow more experimentation as long as there is no "hard compatibility break". The C API has "enabled the Python ecosystem to become the powerhouse that it is", but it is difficult to maintain consistently. He continued:

Those kinds of use cases are more than enough to justify changes to the way we manage our public header files - you don't need to dress it up in "sky is falling" rhetoric founded in the fear of other programming languages. Yes, Python is a nice language to program in, and it's great that we can get jobs where we can get paid to program in it. That doesn't mean we have to treat it as an existential threat that we aren't always going to be the best choice for everything :)

There was general agreement in the thread that reorganizing the header files and API was beneficial. Eric Snow pointed to some work he has done to consolidate the global variables in CPython into a single structure; it could perhaps be used as a starting point for the core API work that Stinner described. Barry Scott, who created PyCXX for writing Python extensions in C++, also liked the idea; he suggested adding a PyCXX-based extension into Stinner's testing regime.

Coghlan posted again, this time looking at more of the details in Stinner's proposal, rather than just the wording in his preamble. He reiterated some of his points about performance not being the best rationale for the initial cleanup work that Stinner is talking about. There is enough confusion in the APIs to justify the cleanup:

We're not sure which APIs other projects (including extension module generators and helper libraries like Cython, Boost, PyCXX, SWIG, cffi, etc) are *actually* relying on. It's easy for us to accidentally expand the public C API without thinking about it, since Py_BUILD_CORE guards are opt-in and Py_LIMITED_API guards are opt-out We haven't structured our header files in a way that makes it obvious at a glance which API we're modifying (internal API, public API, stable ABI)

The guards that Coghlan refers to are supposed to restrict the symbols available to programs; Py_BUILD_CORE is for the interpreter and related tools (effectively what Stinner would put in core ) and Py_LIMITED_API is for the stable ABI (and is badly named according to several in the thread). Coghlan suggested making all of that more clear before tackling further questions:

In particular, better segmenting our APIs into "solely for CPython's internal use", "ABI is specific to a CPython version", "API is portable across Python implementations", "ABI is portable across CPython versions (and maybe even Python implementations)" allows tooling developers and extension module authors to make more informed decisions about how closely they want to couple their work to CPython specifically. And then *after* we've done that API clarification work, *then* we can ask the question about what the default behaviour of "#include <Python.h>" should be, and perhaps introduce an opt-in Py_CPYTHON_API flag to request access to the full traditional C API for extension modules and embedding applications that actually need it.

In the proposal, Stinner said that, instead of including files between the different API directories, declarations should be duplicated in order to avoid mistakes in exposing declarations incorrectly. Duplication has its own set of dangers, however; Coghlan and others in the thread suggested a strict hierarchy of the APIs and their include files such that no duplication was needed but that definitions could not leak out into the other APIs incorrectly.

Along the way, it became clear that the "API" and "ABI" terms were being tossed around without a clear description of what the pieces are. Brett Cannon took a stab at defining the various levels:

The stable A**B**I which is compatible across versions A stable A**P**I which hides enough details that if we change a struct your code won't require an update, just a recompile An API that exposes CPython-specific details such as structs and other details that might not be entirely portable to e.g. PyPy easily but that we try not to break An internal API that we use for implementing the interpreter but don't expect anyone else to use, so we can break it between feature releases (although if e.g. Cython chooses to use it they can)

Coghlan mostly agreed with that, but thought that the portable API (#2 above) should still be able to change over time, subject to the standard Python deprecation policy. He sees the portable API as only exposing interfaces that are genuinely portable to, at least, PyPy. It would also stay as close to the stable ABI as possible, "with additions made *solely* to support the building of existing popular extension modules". So he sees the levels as follows:

stable ABI (strict extension module compatibility policy)

portable API (no ABI stability guarantees, normal deprecation policy)

public CPython API (no cross-implementation portability guarantees)

internal-only CPython core API (arbitrary changes, no deprecation warnings)

While Stinner's motivation may be different from others', it would seem that there is broad agreement that API rationalization is needed. How it all might look at a high level is also fairly non-controversial. An actual PEP that focuses strictly on the API clarification would seem to be the next step. Once that happens, assuming that it does, Stinner and others can start working on ways to make the portable API even more portable in support of various performance optimization experiments.

Comments (none posted)