Anyone who builds an Android app knows that testing on real devices is important. But what if instead of testing on one device at a time, you could test 300,000?

Enter MegaDroid. A project of the US government’s Sandia National Laboratories in California, the aptly named MegaDroid has linked 300,000 Android virtual machines together in a testbed for studying all kinds of network disruptions. Researchers could use MegaDroid for anything they can dream of, but the lab says it envisions projects that seek to “understand and limit the damage from network disruptions due to glitches in software or protocols, natural disasters, acts of terrorism, or other causes.”

MegaDroid, unveiled on Tuesday, is the third in a series of such projects. The first was MegaTux, which booted 1 million Linux kernels as virtual machines in 2009, and MegaWin, which did the same with 100,000 Windows XP and Windows 7 instances. Fritz and team would like to extend the Mega- projects to iOS, but that would require some cooperation on Apple's part.

For MegaDroid, Sandia built a cluster on 520 nodes, each one a quad-core Intel Core i7 processor (Sandy Bridge) with 12GB RAM (and no disk storage), and a Gigabit Ethernet network. It cost about $500,000, a small amount for government labs accustomed to multimillion dollar supercomputers. Sandia actually specced out the cluster through CostCo, but then bought the hardware through a bidding process.

The project is not a simple one. The MegaDroid software—which will be released as open source—can run on anything from a desktop workstation to a high-end supercomputer, said David Fritz, a cyber security researcher in Sandia’s high-performance computing group.

MegaDroid uses an x86 build of Android 4.0, running on qemu-kvm virtual machines. Sandia already figured out how to cluster virtual machines at high density with MegaTux and MegaWin, but the tough part with MegaDroid was creating interfaces to all the sensors in the VMs, so that programs can communicate with the virtual machines just as if they were real devices. That means interacting with the GPS, text messaging, and other capabilities of Android. The virtual machines can also connect to a couple of physical Android phones on T-Mobile’s network.

With GPS interfaces, researchers can feed the Android virtual machines “cooked” mapping data, to emulate hundreds of thousands of people walking around a city. With SMS, “We can actually text the cluster, and the cluster will route the SMS message over an SMS radio we’ve written to the individual VMs and they respond,” Fritz said.

Not every necessary part is well-documented, so “it boiled down to digging into mountains of Android source in their emulator to see how they’d done it and try to emulate that.”

According to Sandia’s announcement, software in the MegaDroid virtual machines treats “location data as indistinguishable from real GPS data, which offers researchers a much richer and more accurate emulation environment from which to analyze and study what hackers can do to smartphone networks.”

Fritz said MegaDroid could be used to test how smartphone networks might be affected by natural disaster. But it could also be used for commercial reasons.

“I imagine somebody like Foursquare would have liked to have a city’s worth of Android devices sitting in the room next door to test their platform on before releasing it into the wild,” he said. “There might be some emergent behavior that you only see at scale.”

Fritz and his team won't be the ones actually using the cluster for science. His role, over the past year, was to get MegaDroid online and build all the appropriate software and networking capabilities to turn it into a testbed that’s ready for researchers.

The software they created will be released under the GPLv3 license before the end of the year. That includes Linux kernel patches and a “wrapper infrastructure” that allows researchers to specify any network topology from local networks to Internet-scale ones.

"The whole point of the project is to bring up these really dense arbitrary network topologies of virtual machines to study whatever behavior we want of large-scale networks," Fritz said. "We can bring up an experiment with a million virtual machines, with an arbitrary network topology, in about ten minutes."

MegaDroid is “only” 300,000 virtual machines strong right now, largely because the Linux-based Android has a lot more code than just the Linux kernel alone. At one point, Fritz’s team got 4.5 million Linux virtual machines running on the Jaguar supercomputer at the Oak Ridge National Laboratory. Currently, 1 million Linux VMs, 100,000 Windows VMs, and 300,000 Android VMs all run on the aforementioned $500,000-cluster at Sandia. They don’t all run at once—think of it as a massive triple-boot machine.

The cluster isn’t in a classified area of Sandia, and may possibly be made available to researchers from outside the lab. But the surest way of the public getting access to MegaDroid capabilities is to get the source code. Government processes move slowly, but Fritz said he expects to have MegaDroid, MegaTux, and MegaWin out as open source by the end of the year. Some components, such as the GPS simulator for MegaDroid, have already been released.

The project still lacks a public website, but Fritz said researchers who want to get what code is available are welcome to contact him at Djfritz@sandia.gov.