[Python-Dev] PEP 466: Network security enhancements for Python 2.7.7

Thanks, Nick. I hereby approve this PEP. You can update the status yourself. Congrats! On Wed, Apr 16, 2014 at 3:04 PM, Nick Coghlan <ncoghlan at gmail.com> wrote: > I've reworded the PEP to make it clear it is now just about > backporting a specific set of enhancements to 2.7.7, as well as > switching to updating to new OpenSSL feature releases in the binary > installers. > > The idea of an open ended backport policy is now listed as a rejected > variant. I believe that change addresses Guido's main remaining > concern, so I think this version is ready for pronouncement. > > Regards, > Nick. > > ======================================== > PEP: 466 > Title: Network Security Enhancements for Python 2.7.7 > Version: $Revision$ > Last-Modified: $Date$ > Author: Nick Coghlan <ncoghlan at gmail.com>, > Status: Draft > Type: Informational > Content-Type: text/x-rst > Created: 23-Mar-2014 > Post-History: 23-Mar-2014, 24-Mar-2014, 25-Mar-2014, 26-Mar-2014, > 16-Apr-2014 > > > Abstract > ======== > > Most CPython tracker issues are classified as errors in behaviour or > proposed enhancements. Most patches to fix behavioural errors are > applied to all active maintenance branches. Enhancement patches are > restricted to the default branch that becomes the next Python version. > > This cadence works reasonably well during Python's normal 18-24 month > feature release cycle, which is still applicable to the Python 3 series. > However, the age of the standard library in Python 2 has now reached a > point > where it is sufficiently far behind the state of the art in network > security > protocols for it to be causing real problems in use cases where upgrading > to > Python 3 in the near term may not be feasible. > > In recognition of the additional practical considerations that have arisen > during the 4+ year maintenance cycle for Python 2.7, this PEP allows a > critical set of network security related features to be backported from > Python 3.4 to the upcoming Python 2.7.7 maintenance release. > > While this PEP does not make any changes to the core development team's > handling of security-fix-only branches that are no longer in active > maintenance, it *does* recommend that commercial redistributors providing > extended support periods for the Python standard library either backport > these features to their supported versions, or else explicitly disclaim > support for the use of older versions in roles that involve connecting > directly to the public internet. > > > New security related features in Python 2.7.7 > ============================================= > > Under this proposal, the following features will be backported from Python > 3.4 to the upcoming Python 2.7.7 maintenance release: > > * in the ``os`` module: > > * persistent file descriptor for ``os.urandom()``. > > * in the ``hmac`` module: > > * constant time comparison function (``hmac.compare_digest()``). > > * in the ``hashlib`` module: > > * password hashing function (``hashlib.pbkdf2_hmac()``). > * details of hash algorithm availability > (``hashlib.algorithms_guaranteed`` > and ``hashlib.algorithms_available``). > > * in the ``ssl`` module: > > * this module is almost entirely synchronised with its Python 3 > counterpart, bringing TLSv1.x settings, SSLContext manipulation, Server > Name Indication, access to platform certificate stores, standard > library support for peer hostname validation and more to the Python 2 > series. > * the only ``ssl`` module features *not* backported under this policy are > the ``ssl.RAND_*`` functions that provide access to OpenSSL's random > number generation capabilities - use ``os.urandom()`` instead. > > As a general change in maintenance policy, permission is also granted to > upgrade to newer feature releases of OpenSSL when preparing the binary > installers for new maintenance releases of Python 2.7. > > This PEP does NOT propose a general exception for backporting new features > to Python 2.7 - every new feature proposed for backporting will still need > to be justified independently. In particular, it will need to be explained > why relying on and independently updated backport on the Python Package > Index > instead is not an acceptable solution. > > > Backwards compatibility considerations > ====================================== > > As in the Python 3 series, the backported ``ssl.create_default_context()`` > API is granted a backwards compatibility exemption that permits the > protocol, options, cipher and other settings of the created SSL context to > be updated in maintenance releases to use higher default security settings. > This allows them to appropriately balance compatibility and security at the > time of the maintenance release, rather than at the time of the original > feature release. > > This PEP does *not* grant any other exemptions to the usual backwards > compatibility policy for maintenance releases. Instead, by explicitly > encouraging the use of feature based checks, it is designed to make it > easier > to write more secure cross-version compatible Python software, while still > limiting the risk of breaking currently working software when upgrading to > a new Python 2.7 maintenance release. > > In all cases where this proposal allows new features to be backported to > the Python 2.7 release series, it is possible to write cross-version > compatible code that operates by "feature detection" (for example, checking > for particular attributes in a module), without needing to explicitly check > the Python version. > > It is then up to library and framework code to provide an appropriate > warning > and fallback behaviour if a desired feature is found to be missing. While > some especially security sensitive software MAY fail outright if a desired > security feature is unavailable, most software SHOULD instead emit a > warning > and continue operating using a slightly degraded security configuration. > > The backported APIs allow library and application code to perform the > following actions after detecting the presence of a relevant > network security related feature: > > * explicitly opt in to more secure settings (to allow the use of enhanced > security features in older maintenance releases of Python with less > secure default behaviour) > * explicitly opt in to less secure settings (to allow the use of newer > Python > feature releases in lower security environments) > * determine the default setting for the feature (this MAY require explicit > Python version checks to determine the Python feature release, but DOES > NOT require checking for a specific maintenance release) > > Security related changes to other modules (such as higher level networking > libraries and data format processing libraries) will continue to be made > available as backports and new modules on the Python Package Index, as > independent distribution remains the preferred approach to handling > software that must continue to evolve to handle changing development > requirements independently of the Python 2 standard library. Refer to > the `Motivation and Rationale`_ section for a review of the characteristics > that make the secure networking infrastructure worthy of special > consideration. > > > OpenSSL compatibility > --------------------- > > Under this proposal, OpenSSL may be upgraded to more recent feature > releases > in Python 2.7 maintenance releases. On Linux and most other POSIX systems, > the specific version of OpenSSL used already varies, as CPython dynamically > links to the system provided OpenSSL library by default. > > For the Windows binary installers, the ``_ssl`` and ``_hashlib`` modules > are > statically linked with OpenSSL and the associated symbols are not exported. > Marc-Andre Lemburg indicates that updating to newer OpenSSL releases in the > ``egenix-pyopenssl`` binaries has not resulted in any reported > compatibility > issues [3]_ > > The Mac OS X binary installers historically followed the same policy as > other POSIX installations and dynamically linked to the Apple provided > OpenSSL libraries. However, Apple has now ceased updating these > cross-platform libraries, instead requiring that even cross-platform > developers adopt Mac OS X specific interfaces to access up to date security > infrastructure on their platform. Accordingly, and independently of this > PEP, the Mac OS X binary installers were already going to be switched to > statically linker newer versions of OpenSSL [4]_ > > > Other Considerations > ==================== > > Maintainability > --------------- > > A number of developers, including Alex Gaynor and Donald Stufft, have > expressed interest in carrying out the feature backports covered by this > policy, and assisting with any additional maintenance burdens that arise > in the Python 2 series as a result. > > Steve Dower and Brian Curtin have offered to help with the creation of the > Windows installers, allowing Martin von Löwis the opportunity to step back > from the task of maintaining the 2.7 Windows installer. > > This PEP is primarily about establishing the consensus needed to allow them > to carry out this work. For other core developers, this policy change > shouldn't impose any additional effort beyond potentially reviewing the > resulting patches for those developers specifically interested in the > affected modules. > > > Security releases > ----------------- > > This PEP does not propose any changes to the handling of security > releases - those will continue to be source only releases that > include only critical security fixes. > > However, the recommendations for library and application developers are > deliberately designed to accommodate commercial redistributors that choose > to apply these changes to additional Python release series that are either > in security fix only mode, or have been declared "end of life" by the core > development team. > > Whether or not redistributors choose to exercise that option will be up > to the individual redistributor. > > > Integration testing > ------------------- > > Third party integration testing services should offer users the ability > to test against multiple Python 2.7 maintenance releases (at least 2.7.6 > and 2.7.7+), to ensure that libraries, frameworks and applications can > still > test their handling of the legacy security infrastructure correctly (either > failing or degrading gracefully, depending on the security sensitivity of > the software), even after the features covered in this proposal have been > backported to the Python 2.7 series. > > > Handling lower security environments with low risk tolerance > ------------------------------------------------------------ > > For better or for worse (mostly worse), there are some environments where > the risk of latent security defects is more tolerated than even a slightly > increased risk of regressions in maintenance releases. This proposal > largely > excludes these environments from consideration where the modules covered by > the exemption are concerned - this approach is entirely inappropriate for > software connected to the public internet, and defence in depth security > principles suggest that it is not appropriate for most private networks > either. > > Downstream redistributors may still choose to cater to such environments, > but they will need to handle the process of downgrading the security > related modules and doing the associated regression testing themselves. > The main CPython continuous integration infrastructure will not cover this > scenario. > > > Motivation and Rationale > ======================== > > The creation of this PEP was prompted primarily by the aging SSL support in > the Python 2 series. As of March 2014, the Python 2.7 SSL module is > approaching four years of age, and the SSL support in the still popular > Python 2.6 release had its feature set locked six years ago. > > These are simply too old to provide a foundation that can be recommended > in good conscience for secure networking software that operates over the > public internet, especially in an era where it is becoming quite clearly > evident that advanced persistent security threats are even more widespread > and more indiscriminate in their targeting than had previously been > understood. While they represented reasonable security infrastructure in > their time, the state of the art has moved on, and we need to investigate > mechanisms for effectively providing more up to date network security > infrastructure for users that, for whatever reason, are not currently in > a position to migrate to Python 3. > > While the use of the system OpenSSL installation addresses many of these > concerns on Linux platforms, it doesn't address all of them (in particular, > it is still difficult for sotware to explicitly require some higher level > security settings). The standard library support can be bypassed by using a > third party library like PyOpenSSL or Pycurl, but this still results in a > security problem, as these can be difficult dependencies to deploy, and > many > users will remain unaware that they might want them. Rather than explaining > to potentially naive users how to obtain and use these libraries, it seems > better to just fix the included batteries. > > In the case of the binary installers for Windows and Mac OS X that are > published on python.org, the version of OpenSSL used is entirely within > the control of the Python core development team, but is currently limited > to OpenSSL maintenance releases for the version initially shipped with the > corresponding Python feature release. > > With increased popularity comes increased responsibility, and this proposal > aims to acknowledge the fact that Python's popularity and adoption is at a > sufficiently high level that some of our design and policy decisions have > significant implications beyond the Python development community. > > As one example, the Python 2 ``ssl`` module does not support the Server > Name Indication standard. While it is possible to obtain SNI support > by using the third party ``requests`` client library, actually doing so > currently requires using not only ``requests`` and its embedded > dependencies, > but also half a dozen or more additional libraries. The lack of support > in the Python 2 series thus serves as an impediment to making effective > use of SNI on servers, as Python 2 clients will frequently fail to handle > it correctly. > > Another more critical example is the lack of SSL hostname matching in the > Python 2 standard library - it is currently necessary to rely on a third > party library, such as ``requests`` or ``backports.ssl_match_hostname`` to > obtain that functionality in Python 2. > > The Python 2 series also remains more vulnerable to remote timing attacks > on security sensitive comparisons than the Python 3 series, as it lacks a > standard library equivalent to the timing attack resistant > ``hmac.compare_digest()`` function. While appropriate secure comparison > functions can be implemented in third party extensions, many users don't > even consider the issue and use ordinary equality comparisons instead > - while a standard library solution doesn't automatically fix that problem, > it *does* make the barrier to resolution much lower once the problem is > pointed out. > > Python 2.7 represents the only long term maintenance release the core > development team has provided, and it is natural that there will be things > that worked over a historically shorter maintenance lifespan that don't > work > over this longer support period. In the specific case of the problem > described in this PEP, the simplest available solution is to acknowledge > that long term maintenance of network security related modules *requires* > the ability to add new features, even while retaining backwards > compatibility > for existing interfaces. > > For those familiar with it, it is worth comparing the approach described in > this PEP with Red Hat's handling of its long term open source support > commitments: it isn't the RHEL 6.0 release itself that receives 10 years > worth of support, but the overall RHEL 6 *series*. The individual RHEL 6.x > point releases within the series then receive a wide variety of new > features, including security enhancements, all while meeting strict > backwards compatibility guarantees for existing software. The proposal > covered in this PEP brings our approach to long term maintenance more into > line with this precedent - we retain our strict backwards compatibility > requirements, but make an exception to the restriction against adding new > features. > > To date, downstream redistributors have respected our upstream policy of > "no new features in Python maintenance releases". This PEP explicitly > accepts that a more nuanced policy is appropriate in the case of network > security related features, and the specific change it describes is > deliberately designed such that it is potentially suitable for Red Hat > Enterprise Linux and its downstream derivatives. > > > Rejected alternative: just advise developers to migrate to Python 3 > ------------------------------------------------------------------- > > This alternative represents the status quo. Unfortunately, it has proven > to be unworkable in practice, as the backwards compatibility implications > mean that this is a non-trivial migration process for large applications > and integration projects. While the tools for migration have evolved to > a point where it is possible to migrate even large applications > opportunistically and incrementally (rather than all at once) by updating > code to run in the large common subset of Python 2 and Python 3, using the > most recent technology often isn't a priority in commercial environments. > > Previously, this was considered an acceptable harm, as while it was an > unfortunate problem for the affected developers to have to face, it was > seen as an issue between them and their management chain to make the case > for infrastructure modernisation, and this case would become naturally > more compelling as the Python 3 series evolved. > > However, now that we're fully aware of the impact the limitations of the > Python 2 standard library may be having on the evolution of internet > security standards, I no longer believe that it is reasonable to expect > platform and application developers to resolve all of the latent defects > in an application's Unicode correctness solely in order to gain access to > the network security enhancements already available in Python 3. > > While Ubuntu (and to some extent Debian as well) are committed to porting > all > default system services and scripts to Python 3, and to removing Python 2 > from its default distribution images (but not from its archives), this is > a mammoth task and won't be completed for the Ubuntu 14.04 LTS release > (at least for the desktop image - it may be achieved for the mobile and > server images). > > Fedora has even more work to do to migrate, and it will take a non-trivial > amount of time to migrate the relevant infrastructure components. While > Red Hat are also actively working to make it easier for users to use more > recent versions of Python on our stable platforms, it's going to take time > for those efforts to start having an impact on end users' choice of > version, > and any such changes also don't benefit the core platform infrastructure > that runs in the integrated system Python by necessity. > > The OpenStack migration to Python 3 is also still in its infancy, and even > though that's a project with an extensive and relatively robust automated > test suite, it's still large enough that it is going to take quite some > time > to migrate fully to a Python 2/3 compatible code base. > > And that's just three of the highest profile open source projects that > make heavy use of Python. Given the likely existence of large amounts of > legacy code that lacks the kind of automated regression test suite needed > to help support a migration from Python 2 to Python 3, there are likely to > be many cases where reimplementation (perhaps even in Python 3) proves > easier than migration. The key point of this PEP is that those situations > affect more people than just the developers and users of the affected > application: the existence of clients and servers with outdated network > security infrastructure becomes something that developers of secure > networked services need to take into account as part of their security > design, and that's a problem that inhibits the adoption of better security > standards. > > As Terry Reedy noted, if we try to persist with the status quo, the likely > outcome is that commercial redistributors will attempt to do something > like this on behalf of their customers *anyway*, but in a potentially > inconsistent and ad hoc manner. By drawing the scope definition process > into the upstream project we are in a better position to influence the > approach taken to address the situation and to help ensure some consistency > across redistributors. > > The problem is real, so *something* needs to change, and this PEP describes > my preferred approach to addressing the situation. > > > Rejected alternative: create and release Python 2.8 > --------------------------------------------------- > > With sufficient corporate support, it likely *would* be possible to create > and release Python 2.8 (it's highly unlikely such a project would garner > enough interest to be achievable with only volunteers). However, this > wouldn't actually solve the problem, as the aim is to provide a *relatively > low impact* way to incorporate enhanced security features into integrated > products and deployments that make use of Python 2. > > Upgrading to a new Python feature release would mean both more work for the > core development team, as well as a more disruptive update that most > potential end users would likely just skip entirely. > > Attempting to create a Python 2.8 release would also bring in suggestions > to backport many additional features from Python 3 (such as ``tracemalloc`` > and the improved coroutine support), making the migration from Python 2.7 > to this hypothetical 2.8 release even riskier and more disruptive. > > This is not a recommended approach, as it would involve substantial > additional work for a result that is actually less effective in achieving > the original aim (which is to eliminate the current widespread use of the > aging network security infrastructure in the Python 2 series). > > Furthermore, while I can't make any commitments to actually addressing > this issue on Red Hat platforms, I *can* categorically rule out the idea > of a Python 2.8 being of any use to me in even attempting to get it > addressed. > > > Rejected alternative: distribute the security enhancements via PyPI > ------------------------------------------------------------------- > > While this initially appears to be an attractive and easier to manage > approach, it actually suffers from several significant problems. > > Firstly, this is complex, low level, cross-platform code that integrates > with the underlying operating system across a variety of POSIX platforms > (including Mac OS X) and Windows. The CPython BuildBot fleet is already set > up to handle continuous integration in that context, but most of the > freely available continuous integration services just offer Linux, and > perhaps paid access to Windows. Those services work reasonably well for > software that largely runs on the abstraction layers offered by Python and > other dynamic languages, as well as the more comprehensive abstraction > offered by the JVM, but won't suffice for the kind of code involved here. > > The OpenSSL dependency for the network security support also qualifies as > the kind of "complex binary dependency" that isn't yet handled well by the > ``pip`` based software distribution ecosystem. Relying on a third party > binary dependency also creates potential compatibility problems for ``pip`` > when running on other interpreters like ``PyPy``. > > Another practical problem with the idea is the fact that ``pip`` itself > relies on the ``ssl`` support in the standard library (with some additional > support from a bundled copy of ``requests``, which in turn bundles > ``backport.ssl_match_hostname``), and hence would require any replacement > module to also be bundled within ``pip``. This wouldn't pose any > insurmountable difficulties (it's just another dependency to vendor), but > it *would* mean yet another copy of OpenSSL to keep up to date. > > This approach also has the same flaw as all other "improve security by > renaming things" approaches: they completely miss the users who most need > help, and raise significant barriers against being able to encourage users > to do the right thing when their infrastructure supports it (since > "use this other module" is a much higher impact change than "turn on this > higher security setting"). Deprecating the aging SSL infrastructure in the > standard library in favour of an external module would be even more user > hostile than accepting the slightly increased risk of regressions > associated > with upgrading it in place. > > Last, but certainly not least, this approach suffers from the same problem > as the idea of doing a Python 2.8 release: likely not solving the actual > problem. Commercial redistributors of Python are set up to redistribute > *Python*, and a pre-existing set of additional packages. Getting new > packages added to the pre-existing set *can* be done, but means approaching > each and every redistributor and asking them to update their > repackaging process accordingly. By contrast, the approach described in > this PEP would require redistributors to deliberately *opt out* of the > security enhancements by deliberately downgrading the provided network > security infrastructure, which most of them are unlikely to do. > > > Rejected variant: provide a "legacy SSL infrastructure" branch > -------------------------------------------------------------- > > Earlier versions of this PEP included the concept of a ``2.7-legacy-ssl`` > branch that preserved the exact feature set of the Python 2.7.6 network > security infrastructure. > > In my opinion, anyone that actually wants this is almost certainly making a > mistake, and if they insist they really do want it in their specific > situation, they're welcome to either make it themselves or arrange for a > downstream redistributor to make it for them. > > If they are made publicly available, any such rebuilds should be referred > to > as "Python 2.7 with Legacy SSL" to clearly distinguish them from the > official > Python 2.7 releases that include more up to date network security > infrastructure. > > After the first Python 2.7 maintenance release that implements this PEP, it > would also be appropriate to refer to Python 2.7.6 and earlier releases as > "Python 2.7 with Legacy SSL". > > > Rejected variant: synchronise particular modules entirely with Python 3 > ----------------------------------------------------------------------- > > Earlier versions of this PEP suggested synchronising the ``hmac``, > ``hashlib`` and ``ssl`` modules entirely with their Python 3 counterparts. > > This approach proved too vague to build a compelling case for the > exception, > and has thus been replaced by the current more explicit proposal. > > > Rejected variant: open ended backport policy > -------------------------------------------- > > Earlier versions of this PEP suggested a general policy change related to > future Python 3 enhancements that impact the general security of the > internet. > > That approach created unnecessary uncertainty, so it has been simplified to > propose backport a specific concrete set of changes. Future feature > backport proposals can refer back to this PEP as precedent, but it will > still be necessary to make a specific case for each feature addition to > the Python 2.7 long term support release. > > > Disclosure of Interest > ====================== > > The author of this PEP currently works for Red Hat on test automation > tools. > If this proposal is accepted, I will be strongly encouraging Red Hat to > take > advantage of the resulting opportunity to help improve the overall security > of the Python ecosystem. However, I do not speak for Red Hat in this > matter, > and cannot make any commitments on Red Hat's behalf. > > > Acknowledgements > ================ > > Thanks to Christian Heimes and other for their efforts in greatly improving > Python's SSL support in the Python 3 series, and a variety of members of > the Python community for helping me to better understand the implications > of the default settings we provide in our SSL modules, and the impact that > tolerating the use of SSL infrastructure that was defined in 2010 > (Python 2.7) or even 2008 (Python 2.6) potentially has for the security > of the web as a whole. > > Thanks to Donald Stufft and Alex Gaynor for identifying a more limited set > of essential security features that allowed the proposal to be made more > fine-grained than backporting entire modules from Python 3.4 [7,8]_. > > Christian and Donald also provided valuable feedback on a preliminary > draft of this proposal. > > Thanks also to participants in the python-dev mailing list threads > [1,2,5,6]_, as well as the various folks I discussed this issue with at > PyCon 2014 in Montreal. > > > References > ========== > > .. [1] PEP 466 discussion (round 1) > (https://mail.python.org/pipermail/python-dev/2014-March/133334.html) > > .. [2] PEP 466 discussion (round 2) > (https://mail.python.org/pipermail/python-dev/2014-March/133389.html) > > .. [3] Marc-Andre Lemburg's OpenSSL feedback for Windows > (https://mail.python.org/pipermail/python-dev/2014-March/133438.html) > > .. [4] Ned Deily's OpenSSL feedback for Mac OS X > (https://mail.python.org/pipermail/python-dev/2014-March/133347.html) > > .. [5] PEP 466 discussion (round 3) > (https://mail.python.org/pipermail/python-dev/2014-March/133442.html) > > .. [6] PEP 466 discussion (round 4) > (https://mail.python.org/pipermail/python-dev/2014-March/133472.html) > > .. [7] Donald Stufft's recommended set of backported features > (https://mail.python.org/pipermail/python-dev/2014-March/133500.html) > > .. [8] Alex Gaynor's recommended set of backported features > (https://mail.python.org/pipermail/python-dev/2014-March/133503.html) > > > > Copyright > ========= > > This document has been placed in the public domain. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20140418/d12d80e3/attachment-0001.html>