Hoping to understand what a new generation of mobile malware could resemble, security researchers will demonstrate a malicious "rootkit" program they've written for Google's Android phone next month at the Defcon hacking conference in Las Vegas.

Once it's installed on the Android phone, the rootkit can be activated via a phone call or SMS (short message service) message, giving attackers a stealthy and hard-to-detect tool for siphoning data from the phone or misdirecting the user. "You call the phone, the phone doesn't ring, and when the phone realizes that it's being called by an attacker's phone number, it sends him back a shell [program]," said Christian Papathanasiou, a security consultant with Chicago's Trustwave, the company that did the research.

Rootkits are stealthy programs designed to cover up their tracks on the operating system in order to evade detection. They have been around on Windows and Unix for years, but lately security researchers have been experimenting with them on mobile platforms.

The hard part of writing an Android rootkit is figuring out how to take advantage of new mobile features while making sure the software runs smoothly on the new platform, Papathanasiou said.

Because the rootkit runs as a module in Android's Linux kernel, it has the highest level of access to the Android phone and can be a very powerful tool for attackers. For example, it could be used to reroute a victim's 911 calls to a bogus number. The rootkit could also track a victim's location or even reroute his browser to a malicious Web site. "Because we interface with the kernel, the opportunities to abuse this are limitless," Papathanasiou said.

On its own, Trustwave's rootkit isn't much of a threat to Android users. That's because a criminal would first need to figure out how to install the software on a victim's phone. This could be done by building the rootkit into a rogue application sold via the Android Market, or by exploiting a new, unpatched bug in Android's Linux kernel that could allow the program to be installed.

Those are pretty big barriers, however.

Google, like other mobile operating system makers, has spent a lot of effort making it hard to get root-level access to the kernel in the first place. "Once someone gets root, the game is essentially up," said Rich Cannings, Android's security leader. "So what we do is prevent people from getting full control of the kernel."

Android has a variety of ways of doing this. It uses application "sandboxing" to prevent one compromised program from gaining access to other parts of the system. It uses other tricks to prevent any undiscovered bugs in the way Android manages its memory from giving hackers a foothold in the system. If the rootkit spreads via the Android Market, Google can get in touch with victims and help them fix the problem.

Security geeks will want to know how the Trustwave team managed to get its rootkit running on Android, but the software won't have much effect on mobile users -- at least for several years -- said Robert Graham, CEO of Errata Security, in Atlanta. Today's mobile phones usually don't ship with good malware detection tools, so developing a rootkit program would be excessive for a typical cybercriminal, he said. One of the main features of a rootkit is that it's very hard to detect, but if the phone isn't looking for malware anyhow, "why not just have your normal malware run on Android?" Graham said.

Malicious software does pop up on mobile devices every now and then, but the vast majority is written for the Windows desktop operating system.

Still, Google's Cannings was careful not to disparage the Trustwave talk. "It's interesting from a theoretical sense that they ported a Linux rootkit to Android," he said. "I think that it helps show that these mobile operating systems are extremely powerful. They're just as powerful as your desktop computer."