Update1: A few hours after this article was published, OpenBSD founder Theo de Raadt emailed Ars and wrote: "It is way overblown. This will never happen in real code." The vulnerability , cataloged as CVE-2014-2970, already has been patched, with modified code located at lib/libc/crypt : arc4random.c.

Update2 on August 1, 2014:Contrary to information de Raadt provided Ars, no CVE was assigned to the bug.

The first "preview" release of OpenSSL alternative LibreSSL is out, and already a researcher says he has found a "catastrophic failure" in the version for Linux.

Edge cases

The problem resides in the pseudo random number generator (PRNG) that LibreSSL relies on to create keys that can't be guessed even when an attacker uses extremely fast computers. When done correctly, the pool of numbers supplied is so vast that the output will almost never be repeated in subsequent requests, and there should be no way for adversaries to accurately predict which numbers are more likely than others to be chosen. Generators that don't produce an extremely large pool of truly random numbers can undermine an otherwise robust encryption scheme. The Dual EC_DRBG influenced by the National Security Agency and used by default in RSA's BSAFE toolkit, for instance, is reportedly so predictable that it can undermine the security of applications that rely on it

A security researcher has warned that there are cases where the LibreSSL PRNG will produce identical output two or more times when running on Linux systems, something he called a "catastrophic failure." The same data can be returned when an application process is cloned—or "forked," in computing parlance—something that can happen when an operating system repeats a similar task over and over, like each time a Web server opens a new connection, for example. In most cases, LibreSSL will detect that a process has been forked because its identifier, known as a PID, will differ. In those cases, LibreSSL will automatically reseed the random numbers to ensure they're unique to the new process.

The failure results in cases where the same 16-bit PID is used to designate two or more processes. Linux ensures that a process can never have the same ID as the child process it spawned, but it remains possible for a process to have the same PID as its grandparent process. The condition appears to be an edge case, but it's one that may be possible if the Linux fork_rand program forks enough times to produce identical PIDs. OpenSSL, the open-source program LibreSSL aims to replace, has ways to recover from such cases. LibreSSL does not, at least not on Linux.

"You may think that fork_rand is a contrived example or that it's unlikely in practice for a process to end up with the same PID as its grandparent," Linux developer Andrew Ayer wrote in a blog post published over the weekend. "You may be right, but for security-critical code, this is not a strong enough guarantee. Attackers often find extremely creative ways to manufacture scenarios favorable for attacks, even when those scenarios are unlikely to occur under normal circumstances.

A related weakness happens when the LibreSSL PRNG runs inside a highly restricted process known as a "chroot jail." Administrators set up such processes to make them harder to be hacked. In the event that LibreSSL is running in such a process, it usually will be unable to access the PRNG resources located in the /dev/urandom directory. In those cases, LibreSSL will look for random data in the now-retired sysctl call. Failing that, it falls back to getting random bits from sources, including the PID, the time of day, memory address, and other predictable properties of the running process. OpenSSL, by contrast, has the ability to detect and better respond to such cases.

Again, the conditions necessary for either of these catastrophic failures don't appear to be easily met in most practical situations, but they do remain possible. And as Ayer noted, hackers often succeed in figuring out ways to force a targeted system to carry out tasks in highly unlikely ways when they make exploitation possible. Still the threat stemming from hacks is serious. In the case of certain cryptographic algorithms, when a system signs two different messages with the same random bits, an attacker can drive the private key.

Reducing the OpenSSL bloat

One of the chief criticisms of OpenSSL is that its code base has grown so big that it's not possible for anyone to fully understand how it behaves. Its support of so many platforms, and so many options on each platform, offers convenience, but it also creates a complexity that can result in the kind of catastrophic hacks that followed the April disclosure of the Heartbleed vulnerability. second OpenSSL fork known as BoringSSL has been spun off by Google developers.

LibreSSL developers aim to improve things by repurposing the most crucial functions of OpenSSL while eliminating some lesser used functions. The cases presented by Ayer underscore the double-edged blade of reducing the OpenSSL bloat. While it will almost certainly improve security, it may also remove security checks that guard against certain types of risks. Ars was unsuccessful in reaching LibreSSL developers for comment. In fairness to them, the program is available only in preview releases that by definition aren't fit for production environments. Ayer's post provides precisely the kind of feedback these developers are likely seeking as they work to secure their OpenSSL fork.

"The problem is that there's no 'right' or 'wrong' here," Matt Green, a Johns Hopkins professor specializing in cryptography, wrote in an e-mail to Ars. "It's like asking whether the crew of the Space Shuttle should have parachutes to protect them in the event that the shuttle explodes. OpenSSL says 'yes,' LibreSSL says a better idea would be to prevent the shuttle from exploding in the first place. Very frustrating to referee that one. All that said: these are real issues and will bite people."