Updated Multiple servers used to maintain and distribute the Linux operating system were infected with malware that gained root access, modified system software, and logged passwords and transactions of the people who used them, the official Linux Kernel Organization has confirmed.

The infection occurred no later than August 12 and wasn't detected for another 17 days, according to an email John "'Warthog9" Hawley, the chief administrator of kernel.org, sent to developers on Monday. It said a trojan was found on the personal machine of kernel developer H Peter Anvin and later on the kernel.org servers known as Hera and Odin1. A secure shell client used to remotely access servers was modified, and passwords and user interactions were logged during the compromise.

“Intruders gained root access on the server Hera,” kernel.org maintainers wrote in a statement posted to the site's homepage shortly after Hawley's email was leaked. “We believe they may have gained this access via a compromised user credential; how they managed to exploit that to root access is currently unknown and is being investigated.”

The maintainers said they believed the repositories used to store Linux source code were unaffected by the breach, although they said they were in the process of verifying its security. They went on to say the potential damage that can be done by rooting kernel.org is less than typical software repositories because of safeguards built in to the system.

“For each of the nearly 40,000 files in the Linux kernel, a cryptographically secure SHA-1 hash is calculated to uniquely define the exact contents of that file,” the statement explained. “Once it is published, it is not possible to change the old versions without it being noticed.”

Each hash is stored on thousands of different systems all over the world, making it easy for users to check the validity of Linux files before running them on their machines.

Linux kernel maintainers didn't respond to an email seeking comment for this story, but two security researchers who were briefed on the breach said the infected systems were hit by a self-injecting rootkit known as Phalanx, variant of which has attacked sensitive Linux systems before.

“It's sort of surprising,” said Jon Oberheide, one of the Linux security researchers briefed on the breach. “If this was a very sophisticated attack, it's very unlikely that the attackers would use an off-the-shelf rootkit like Phalanx. Normally if you were to target a high-value target you would potentially use something that's more more tailored to your specific target, something that's not going to be flagged or potentially detected.”

Fellow security researcher Dan Rosenberg said he was also briefed that the attackers used Phalanx to compromise the kernel.org machines. Both Rosenberg and Oberheide confirmed that Hawley's email was sent to Linux kernel developers. It was also signed using Hawley's private encryption key.

The first indication of a compromise came shortly after an error message related to Xnest was displayed on a machine that didn't have the X Window application installed. Linux maintainers are advising developers to carefully investigate any systems that don't have the the program installed and display the /dev/mem message anyway.

Been down this road before

It's not the first breach to hit a venerable organization that distributes open-source software that thousands of sensitive organizations rely on to remain secure. In December, GNU Savannah, the main source-code repository for the Free Software Foundation, was taken down following a hack that compromised passwords. Admins at the time couldn't rule out the possibility the attackers gained root access.

And in April 2010, the Apache Software Foundation, which maintains the world's most widely used webserver, suffered a direct targeted attack that captured he passwords of anyone who used the website's bug-tracking service over a three-day span. It was the second major compromise of Apache.org in eight months.

Kernel.org members have taken the infected servers offline and are in the process of completely reinstalling the operating system on each machine in the organization. They are also working with all 448 users of kernel.org to change their authentication credentials, including SSH keys. They have also notified authorities in the US and Europe to assist in the ongoing probe of the breach.

“The Linux community and kernel.org take the security of the kernel.org domain extremely seriously, and are pursuing all avenues to investigate this attack and prevent future ones,” Wednesday's statement said. ®