The open-source Linux operating system contains a serious security flaw that can be exploited to gain superuser rights on a target system.

The vulnerability, in the Linux implementation of the Reliable Datagram Sockets (RDS) protocol, affects unpatched versions of the Linux kernel, starting from 2.6.30, where the RDS protocol was first included.

According to VSR Security, the research outfit that discovered the security hole, Linux installations are only vulnerable if the CONFIG_RDS kernel configuration option is set, and if there are no restrictions on unprivileged users loading packet family modules, as is the case on most stock distributions.

Because kernel functions responsible for copying data between kernel and user space failed to verify that a user-provided address actually resided in the user segment, a local attacker could issue specially crafted socket function calls to write arbritrary values into kernel memory. By leveraging this capability, it is possible for unprivileged users to escalate privileges to root.

The company has released a proof-of-concept exploit to demonstrate the severity of the vulnerability. The folks at The H Security tested the exploit on Ubuntu 10.04 (64-bit) and successfully opened a root shell.

A fix for this issue has been committed by Linus Torvalds. VSR Security recommends that users install updates provided by downstream distributions or apply the committed patch and recompile their kernel.