All Things Pythonic

It isn't Easy to Remove the GIL

by Guido van van Rossum

September 10, 2007



Summary

A response to a blog post by Juergen Brendel pleading for the removal of the GIL.


yesterday, Juergen Brendel blogs at length about the disadvantages of the GIL. He claims it is an architectural decision I'm making that limits his productivity.

I don't expect that this or any response will stop the requests for the GIL's removal, despite a well-reasoned FAQ entry about the issue. But I also don't expect it to go away until someone other than me goes through the effort of removing it, and showing that its removal doesn't slow down single-threaded Python code.

This has been tried before, with disappointing results, which is why I'm reluctant to put much effort into it myself. In 1999 Greg Stein (with Mark Hammond?) produced a fork of Python (1.5 I believe) that removed the GIL, replacing it with fine-grained locks on all mutable data structures. He also submitted patches that removed many of the reliances on global mutable data structures, which I accepted. However, after benchmarking, it was shown that even on the platform with the fastest locking primitive (Windows at the time) it slowed down single-threaded execution nearly two-fold, meaning that on two CPUs, you could get just a little more work done without the GIL than on a single CPU with the GIL. This wasn't enough, and Greg's patch disappeared into oblivion. (See Greg's writeup on the performance.)

I'd welcome it if someone did another experiment along the lines of Greg's patch (which I haven't found online), and I'd welcome a set of patches into Py3k only if the performance for a single-threaded program (and for a multi-threaded but I/O-bound program) does not decrease.

I would also be happy if someone volunteered to maintain a GIL-free fork of Python, in case that the single-threaded performance goal can't be met but there is significant value for multi-threaded CPU-bound applications. We might even end up with all the changes permanently part of the code base, but enabled only on request at compile time.

However, I want to warn that there are many downsides to removing the GIL. It complicates life for extension modules, who can no longer expect that they are invoked in a "safe zone" protected by the GIL -- as soon as an extension has any global mutable data, will have to be prepared with concurrent calls from multiple threads. There might also be changes in the Python/C API necessitated by the need to lock certain objects for the duration of a sequence of calls.

While it is my personal opinion, based upon the above considerations, that there isn't enough value in removing the GIL to warrant the effort, I will welcome and support attempts to show that times have changed. However, there is no point in pleading alone -- Python is open source and I have my hands full dealing with the efforts to produce a quality 3.0 language definition and implementation on time. I want to point out one more time that the language doesn't require the GIL -- it's only the CPython virtual machine that has historically been unable to shed it.

Talk Back!

Have an opinion? Readers have already posted 54 comments about this weblog entry. Why not add yours?

RSS Feed

If you'd like to be notified whenever Guido van van Rossum adds a new entry to his weblog, subscribe to his RSS feed.

About the Blogger

Guido van Rossum is the creator of Python, one of the major programming languages on and off the web. The Python community refers to him as the BDFL (Benevolent Dictator For Life), a title straight from a Monty Python skit. He moved from the Netherlands to the USA in 1995, where he met his wife. Until July 2003 they lived in the northern Virginia suburbs of Washington, DC with their son Orlijn, who was born in 2001. They then moved to Silicon Valley where Guido now works for Google (spending 50% of his time on Python!).

This weblog entry is Copyright © 2007 Guido van van Rossum. All rights reserved.