Gubrud, an accomplished academic, first proposed a ban on autonomous weapons back in 1988. He’s typically polite, but talk of robotics brings out his combative side: he approached DARPA director Arati Prabhakar at one point during the challenge and tried to get her to admit that the agency is developing autonomous weapons.

He may have been the lone voice of dissent among the hundreds of robot-watchers at DARPA’s event, but Gubrud has some muscle behind him: the International Committee for Robot Arms Control (ICRAC), an organization founded in 2009 by experts in robotics, ethics, international relations, and human rights law. If robotics research continues unchecked, ICRAC warns, the future will be a dystopian one in which militaries arm robots with nuclear weapons, countries start unmanned wars in space, and dictators use killer robots to mercilessly control their own people.

Autonomous robots by design

Humanoid Most robotic weapons in the military don’t have hands or faces; they tend to look more like airplanes or small tanks. Android soldiers are still unnerving to imagine, though, because they do resemble people. Such robots are useful in disaster response because rescue takes place in human environments, DARPA says. But robots that can walk around in post-earthquake rubble would also be suited for combat in places like Baghdad and Kandahar. Underwater It’s harder to accidentally kill civilians when you’re underwater, and for that reason we may see autonomous weapons proliferate there before we see them on land. Autonomous undersea weapons could range from sea mines and torpedoes to submarines known as autonomous underwater vehicles (AUVs). Radio signals also don’t travel well in water, creating an incentive for programming greater autonomy. Land The most autonomous weapons currently in use are land-defense systems that react to incoming threats. Examples include the "close-in weapons systems" employed by the US to shoot down missiles, or the Samsung Techwin SGR-A1, which is replacing human guards along parts of the South Korean border. That robot detects when a person enters its range and asks for a password. If a person offers the wrong one, the SGR-A1 can be set to fire automatically.

Concern about robot war fighters goes beyond a "cultural disinclination to turn attack decisions over to software algorithms," as the autonomy hawk Barry D. Watts put it. Robots, at least right now, have trouble discriminating between civilians and the terrorists and insurgents who live among them. Furthermore, a robot’s actions are a sum of its programmer, operator, manufacturer, and other factors, making it difficult to assign responsibility if something does go wrong. And finally, replacing soldiers with robots would convert the cost of war from human lives to dollars, which could lead to more conflicts.

ICRAC and more than 50 organizations including Human Rights Watch, Nobel Women’s Initiative, and Code Pink have formed a coalition calling itself the Campaign to Stop Killer Robots. Their request is simple: an international ban on autonomous weapons systems that will head off the robotics arms race before it really gets started.

"Tireless war machines, ready for deployment at the push of a button, pose the danger of permanent ... armed conflict."

There has actually been some progress on this front. A United Nations report in May, 2013 called for a temporary ban on autonomous lethal systems until nations set down rules for their use. "There is widespread concern that allowing lethal autonomous robots to kill people may denigrate the value of life itself," the report says. "Tireless war machines, ready for deployment at the push of a button, pose the danger of permanent (if low-level) armed conflict."

The UN Convention on Certain Conventional Weapons will convene a meeting of experts this spring, the first step toward an international arms agreement. "We need to have a clear view of what the consequences of those weapons could be," says Jean-Hugues Simon-Michel, the French ambassador to the UN Conference on Disarmament and its chairman, who persuaded the other nations to take up the issue. "And of course when there is a particular concern with regard to a category of weapons, it’s always easier to find a solution before those weapons exist."

Watching the robots stumble around the simulated disaster areas at the DARPA trials would have been reassuring to anyone worried about killer robots. Today’s robots are miracles of science compared to those from 20 years ago, but they are still seriously impaired by lousy perception, energy inefficiency, and rudimentary intelligence. The machines move agonizingly slowly and wear safety harnesses in case they fall, which happens often.

The capabilities being developed for the challenge, however, are laying the groundwork for killer robots should we ever decide to build them. "We’re part of the Defense Department," DARPA’s director, Arati Prabhakar, acknowledges. "Why do we make these investments? We make them because we think that they’re going to be important for national security." One recent report from the US Air Force notes that "by 2030 machine capabilities will have increased to the point that humans will have become the weakest component in a wide array of systems and processes."

"If we can protect innocent civilian life, I do not want to shut the door on the use of this technology."

By some logic, that might be a good thing. Robot shooters are inherently more accurate than humans, and they’re unaffected by fear, fatigue, or hatred. Machines can take on more risk in order to verify a target, loitering in an area or approaching closer to confirm there are no civilians in the way.

"If we can protect innocent civilian life, I do not want to shut the door on the use of this technology," says Ron Arkin, PhD, a roboticist and ethicist at the Georgia Institute of Technology who has collaborated extensively with Pentagon agencies on various robotics systems.

Arkin proposes that an "ethical governor," a set of rules that approximates an artificial conscience, could be programmed into the machines in order to ensure compliance with international humanitarian law. Autonomy in these systems, he points out, isn’t akin to free will — it’s more like automation. During the trials, DARPA deliberately sabotaged the communications links between robots and their operators in order to give an advantage to the bots that could "think" on their own. But at least for now, that means being able to process the command "take a step" versus "lift the right foot 2 inches, move it forward 6 inches, and set it down."

"When you speak to philosophers, they act as if these systems will have moral agency," Arkin says. "At some level a toaster is autonomous. You can task it to toast your bread and walk away. It doesn’t keep asking you, ‘Should I stop? Should I stop?’ That’s the kind of autonomy we’re talking about."