A long time ago, in the 1980s, a coven of Berkeley sorcerers crafted an operating system. They named it after themselves: the Berkeley Software Distribution, or BSD. For generations they nurtured it, growing it and adding features. One night, they conjured a powerful function that could resolve hostnames to IPv4 or IPv6 addresses. It was called getaddrinfo. The function was mighty, but in years to come it would grow dangerous, for the sorcerers had not made getaddrinfo thread-safe.

As ages passed, BSD spawned many offspring. There were FreeBSD, OpenBSD, NetBSD, and in time, Mac OS X. Each made its copy of getaddrinfo thread safe, at different times and different ways. Some operating systems retained scribes who recorded these events in the annals. Some did not.

Because getaddrinfo is ringed round with mystery, the artisans who make cross-platform network libraries have mistrusted it. Is it thread safe or not? Often, they hired a Mutex Troll to stand guard and prevent more than one thread from using getaddrinfo concurrently. The most widespread such library is Python's own socket module, distributed with Python's standard library. On Mac and other BSDs, the Python interpreter hires a Mutex Troll, who demands that each Python thread hold a special lock while calling getaddrinfo.

Behold, my friends, the getaddrinfo lock in Python's socketmodule.c:

/* On systems on which getaddrinfo() is believed to not be thread-safe, (this includes the getaddrinfo emulation) protect access with a lock. */ #if defined(WITH_THREAD) && (defined(__APPLE__) || \ (defined(__FreeBSD__) && __FreeBSD_version+0 < 503000) || \ defined(__OpenBSD__) || defined(__NetBSD__) || \ defined(__VMS) || !defined(HAVE_GETADDRINFO)) #define USE_GETADDRINFO_LOCK #endif #ifdef USE_GETADDRINFO_LOCK #define ACQUIRE_GETADDRINFO_LOCK PyThread_acquire_lock(netdb_lock, 1); #define RELEASE_GETADDRINFO_LOCK PyThread_release_lock(netdb_lock); #else #define ACQUIRE_GETADDRINFO_LOCK #define RELEASE_GETADDRINFO_LOCK #endif

This lock was not widely known. Although Python's Global Interpreter Lock certainly is infamous, the getaddrinfo lock was known only to a battle-worn few. The Mutex Troll required this lock in Python interpreters installed on Mac, NetBSD, OpenBSD, or on FreeBSD before 5.3. I first descried it while hunting a deadlock it caused in PyMongo. Since then, the mercenary troll and I had met in combat again and again: deadlocks, errors, and slowdowns in my Python code led me to renewed confrontation with it.

As I met more Python experts, I learned that they had encountered this hired troll, too. For example, multithreaded Python code that crawls thousands of websites, and must resolve thousands of hosts, ran fine on Linux but came to grief on a Mac. Threads would wait in a long queue to acquire the lock before the troll guard would allow them to call getaddrinfo. One very slow DNS resolution would block all the threads behind it, and they would throw timeouts before they could ever grasp the lock.

They raised their flagons of Diet Coke and drank to my good fortune.

The day that Python's artisans hired the Mutex Troll it was needed to safeguard getaddrinfo against concurrent threads; but now the troll was no longer needed. I knew that getaddrinfo had been made thread safe on BSD's children, especially the most famous of them: Mac OS X. Many modern programs that call getaddrinfo concurrently suffer no harm. The MongoDB server, for example, runs fine on Mac without a getaddrinfo lock nor a troll to enforce it. But the mercenary's contract was eternal, and in the decades it stood guard over the lock it had grown corrupt and greedy. The time had come to banish the horrid thing. Whenever I read that comment from some past craftsman about "systems on which getaddrinfo() is believed to not be thread-safe", my ire boiled hotter. Why enthrall ourselves to mere belief, not knowing the truth?

One winter morning last year, I stood before my companions in the daily status meeting and asked leave to endeavor on a quest. I told them about the Mutex Troll and how it had held Mac and BSD coders hostage for generations. I made a great boast: I would defeat the Mutex Troll in Python and free the threads. Gladly my fellows at MongoDB granted me leave to go on the journey. "Banish the troll for the good of all!" they cried. They raised their flagons of Diet Coke and drank to my good fortune.

I donned my war-gear and sallied from MongoDB's castle. But to dispel the Mutex Troll's power in Python, it is not enough to say "perchance getaddrinfo once was broken, but now it is surely mended". When was getaddrinfo fixed, and how? And how could I prove it to the Python core developers? These developers, unlike MongoDB coders, must support all ancient versions of OS X to the dawn of time. To convince them, I would need to know the answer for certain. I decided to ask an Apple engineer to aid my cause.

Apple engineers are not like you and me — they are a shy and secretive folk. They publish only what code they must from Darwin. Their comings and goings are recorded in no bug tracker, their works in no changelog. To learn their secrets, one must delve deep.

I wept bitterly over the years of needless toil that programmers and processors had suffered.

Through wild hills I journeyed to a tower where Apple clerics once gathered. I entered the deserted tower and found carved into the wall a man page for getaddrinfo on OS X 10.4, which warned:

getaddrinfo(3) BSD Library Functions Manual BUGS: The implementation of getaddrinfo() is not thread-safe. December 20, 2004

I read the source for 10.4's getaddrinfo. Uncertain what I beheld, I guessed I saw the data race: getaddrinfo calls gai_lookupd , which reads and writes a global static variable gai_proc . It seemed ill-wrought for multithreading.

On OS X 10.5, the warning had vanished from the man page, and the getaddrinfo function was largely rewritten. Should I believe that the bug was fixed then, a decade ago? I wept bitterly over the years of needless toil that programmers and processors had suffered at the hands of the troll. I pitied them, but I did not falter. I would prove that they were free of the troll's domination. Yet, diffing one version of getaddrinfo to the next was unprofitable. I did not understand what I saw! I needed an answer from Apple.

To ask a question of the Apple engineers, my friends, you must leave $99 of silver coins in a hollow oak tree. Then, wait. It may take a day, or a season, but an Apple engineer will come and whisper in your ear, and bind you to a secret pact that you must never reveal what you have been told. The engineer will give you an Asking Ring. This you must use to ask a second question within a year and a day, or its power is lost.

I returned to MongoDB and asked my companions for some silver coins, which they gave me gladly. Then, on the first night after the first day of the year, I left them in the hollow oak, with my question:

"Has getaddrinfo been fixed? Can you give me a public statement or a link to a resolved bug in a tracker? I need a way, not only to know it was fixed, but to prove it to others."

I did not yet know what my second question would be.

Twelve days and twelve nights I waited, refreshing my email. Is today the day? Or today? The twelfth morning, January 13, I awoke to see an ancient box, of rusted hinge and hoared with lichen, resting by my bed. The box opened, exhaling the dust of forgotten smithies where the first network code was forged. Slowly, I reached in. I lifted out a scroll marked with assembly codes and unfurled it with a crackle.