SCHAFT, A BLUE-LIMBED robot, lifts its right foot to the seventh step of the ladder, its left foot to the eighth, and stops; it sways alarmingly in the strong Florida sea breeze. Of the 17 teams competing in the DARPA Robotics Challenge (DRC), a first-of-its-kind event held at a speedway track near Miami in December 2013, only two others got their robots this high up the ladder. One of those two then took a nasty tumble.

For most of a minute SCHAFT is still, except for a flap on its chest that slowly rises and falls in a breathing motion. Then it springs into action again. Its left knee straightens, its right foot rises, its left knee bends again—not forwards, as a human knee would, but backwards—and in four swift movements it firmly plants both feet on the platform at the top of the ladder.

With this latest triumph SCHAFT has become the undisputed champion of the DRC. In the past two days it has driven a small jeep-like car over a short, twisting course, walked over ramps, steps and rubble, negotiated various doorways, cleared debris from its path, cut a hole in a wall with a power tool, connected a fire hose and shut off a series of valves. Now, as the Japanese engineers who built it celebrate below, it squats impassively on its backwards-facing haunches.

The prize for victory is not just the applause of fans, rivals and robo-curious spectators, who have come in their thousands. DARPA, the Pentagon research agency which runs the DRC, is rewarding the best teams at the event with up to $1m so that they can improve their robots and compete again in a year’s time at a more demanding second event. All told the project is costing it some $80m.

The agency made robots a priority because, like many others, it suspects that the technology may be on the cusp of scaling far greater heights than a nine-step aluminium ladder. It is expressing its support in the unusual, quasi-sporting, highly public forum of the DRC because robotics is a technology unlike any other. As machines that sense their environment, analyse it and respond accordingly, robots lend themselves to showmanship, judged as they are by their actions in the world (this special report will deal only glancingly with other machines sometimes called robots that do not have a moving physical presence, such as software “bots” or stationary bits of automation). They exert a fascination, both on their designers and their fans, that transcends the technology’s current practical uses. The engineers who made SCHAFT started their company not because they thought it would make them a fortune, says Takashi Kato, an entrepreneurial investor who helped them with it; they did it “because they would rather build robots than anything else”.

This fascination has produced robots of many shapes and sizes. Academics have tried their hand at mimicking nature, basing robots on everything from termites to pterodactyls. For robots designed to make money, form has followed function, leading to the multi-jointed, mostly cast-iron arms of the world’s 1.2m-1.5m manufacturing robots; the spindlier limbs of robots designed for surgery; the deep-pan pizza-dish form of service robots that vacuum the floors of the house-proud and gadget-friendly. But at the DRC, as in the public imagination, the robots are mostly humanoid.

There are exceptions. RoboSimian, competing on behalf of JPL, the laboratory that runs most of NASA’s planetary missions, looks appropriately alien, with the knees, or elbows, of its four limbs articulated in ways that are distinctly non-human, and for that matter un-simian; it moves more like a body-popping spider. SCHAFT, though, has two legs and two arms, even if it lacks a recognisable head and its hips do double duty as shoulders. Hubo, a South Korean robot being used by two of the teams, and Atlas, the machine chosen by seven American teams, go the whole arms-legs-head-and-shoulders humanoid hog.

The reason for this convergence on the humanoid form is that they must function in an environment shaped to human specifications. The ladders, doors, valves and rubble of the DRC are meant to test how well robots would cope if sent into a disaster area inaccessible to humans, such as a stricken nuclear power station or chemical plant. And although such rescue operations are unusual, the constraints they impose fit with one of the main aims of current robotic research: learning how to operate flexibly in an environment designed for humans, not robots.

Inspect the complete line-up here Not coincidentally, such operations are typical of robots in science fiction—the land of their birth. More thoroughly than any other technology—except, perhaps, that of the spaceship—robots were imagined in print and on film long before they were created in laboratories and installed in factories (some of the cinematic imaginings serve to illustrate this special report). And to some extent robots remain science fiction to this day; they may have become real, but they continue to be shaped by expectations created by fiction and continuously nurtured by it. There have always been stories of artificial people and magical mechanisms. They were, though, for the most part singular creations, given their being through mystical powers or masterly skills. The fiction of the 20th century introduced something new: mass production. Like the factory-made labourers to which the word was first applied in Karel Capek’s play, “R.U.R.: Rossum’s Universal Robots”, robots became both the products of industrial technology and a way of talking about that technology’s effects: of what it does to people’s futures; of how it can make them robotic in themselves; of how it still always seems to leave some space for the strange and quirky.

Fiction before fact

Capek’s factory-born robots, embodying anxieties about industrial progress, rose up to wipe out the human race, and many of their fictional successors have followed a similar course. But Isaac Asimov, a Russian-born American who did more than anyone else to steer science fiction towards the idea of robots as industrial artefacts, offered a kinder, more complex version of the conflict between the made and the makers. Before electronic computers existed, Asimov saw that robots would be programmed, and thus constrained by their programming. He also realised that humans would fail to appreciate the predictability such programming brought, and that the clash between what was programmed and what humans expected of, wanted from and feared in their robots would be a rich source of plots.

But for all that they were industrial, Asimov’s robots were also the product of a particular sensibility, background and set of concerns—those of a child of hard-working and hard-pressed Russian parents in 1930s Brooklyn. Always content to do what they are told; always consigned to work on the “dull, dirty, dangerous” jobs; often uneasily aware that they are superior in some ways to their masters; endlessly at risk of pogrom because of their masters’ resentment and fear of them: his robot stories, and those of his successors, were immigrant stories. Except that the robots are immigrants not from abroad but from the future.

Robot researchers are keenly aware of the fictional foundations of their work. Gill Pratt, an academic from the Massachusetts Institute of Technology (MIT) currently on secondment to DARPA, where he runs the DRC programme, immediately brings up Asimov when asked why he got interested in robots. Any visit to a Japanese robot laboratory soon leads to a discussion about Astro Boy, the helpful android who in the 1960s starred in Japan’s first popular animated television show, to help explain the country’s rampant robophilia. And robots that offer domestic services are routinely compared to Rosie, the robot maid in “The Jetsons”, an American television show of the same vintage. No discussion of the military use of drones will continue for long without reference to the “Terminator” films.

Yet those who work with robots also know better than anyone else that what they do, although prefigured and even shaped by fiction, still falls far short of it. Willow Garage, a robotics company founded in 2006 by Scott Hassan, one of the first people to work at Google, spent millions of dollars developing PR2, a two-armed “personal robot” designed to help with tasks at home and elsewhere. Able to navigate itself and manipulate objects of various sorts with its hands, it is about as good at what it does as any robot built so far, and dozens have been sold or donated to research laboratories around the world. Still, it is, Mr Hassan says, “dumber than a doornail”.

In “A Christmas Carol”, the first thing Charles Dickens tells the reader is that Jacob Marley is “dead as a doornail”: this fact “must be distinctly understood, or nothing wonderful can come of the story I am to relate”. Something similar applies to the doornail dumbness of robots. To see what may come of them, however wonderful, you have distinctly to understand how very little thought they are currently capable of. They are roughly as intelligent as a small bug, says Mr Pratt.

The field of artificial intelligence (AI), from which academic robotics has developed, has achieved quite a lot since it was founded in the 1960s—but nothing like the generalised intelligence, capable of seeing, understanding and planning, that those founders were after. It has shown that although computers can easily do some things people find hard (such as playing chess), they cannot fathom many things people can do without thinking. Getting robots to walk moderately well has taken decades and many hundreds of millions of dollars, mostly spent in Japan. To get a (non-walking) PR2 simply to recognise a thing that needs picking up still takes a lot of work. In Masayuki Inaba’s laboratory at the University of Tokyo, where some of the SCHAFT team got their start, a PR2 programmed by gifted students tried to serve your correspondent a can of coffee from a fridge. It opened the fridge door and got the coffee out, but then tried to serve the can to the fridge instead.

The reason why a robot like SCHAFT can negotiate doors, climb ladders, cut holes in walls and so on is that it is getting help from humans. All the robots at the DRC were being “tele-operated”; their near-term goals were set and monitored by operators in the garages that line the speedway track’s pit lane. The robots were keeping their balance and taking their steps using on-board software and processing power; the back-room boys were interpreting what the robots saw and planning their next moves.

The robots that did best in Florida will reconvene in late 2014 or early 2015 for the finals, where the tasks will be harder and performance, everyone hopes, better. Mr Pratt explains that one purpose of the DARPA challenge is to give a sense of how much robotic progress a year of research and funding can buy. A previous competition proved wildly successful at promoting progress in a related field. The first DARPA Grand Challenge, in 2004, required teams to get cars to drive themselves over a 240km desert route. None of them made it even a 20th of the distance. Yet when the race was reconvened a year later, better software for mapping and understanding the world allowed five competitors to complete the course. In the exhibition ground at the DRC sat one of Google’s driverless cars, its ancestry directly traceable to the winning team in that second challenge. If such progress is possible on the roads, why not in the kitchen, the retirement home or the shopping mall?

The comparison is given extra bite by Google’s acquisition, in the run-up to the DRC, of eight robotics companies with products and services at various levels of development. They included the Japanese startup that made SCHAFT as well as Boston Dynamics, which has done a great deal of work for DARPA. It designed and built the Atlas robots that most of the American DRC teams were using, and has produced impressive walking and running quadruped robots for military test programmes.

Putting money on it

Google is being tight-lipped about its plans for all these robots; speculation on what lies ahead ranges from far better factory automation and door-to-door delivery robots to a mission to the moon which will allow, on the 50th anniversary of Apollo 11, one small step for robotkind. But the mere fact that a company with an impressive track record in innovation has rounded up a lot of robot engineering talent and intellectual property is a striking vote of confidence in the field’s prospects. That does not mean that robots in general will progress as quickly as driverless cars have done; progress in robotics will often be limited by the rate at which the most difficult of all the different kinds of problem encountered can be solved. But change does not have to be fast for its long-run consequences to be profound.

Nor do robots have to become fully autonomous in order to be able to do a far wider range of interesting and useful things than they do today. For the most part they are not replacements for humans; they are better seen as extensions. Humans can come together to do things they cannot do alone; in future they will increasingly come together with robots to do things they cannot otherwise do so easily, or in some cases at all.

Robotic extensions will come in a variety of forms. But just as the need to work at a range of tasks in the human world can force robots into a humanoid shape, so the need to work with humans in that world means that many of them will become, to some extent, socially humanoid. The most useful robots will be those that are best suited to working with people, at whatever level of autonomy is appropriate to the task. To fit into the social world, robots will need to take both casual and formal instructions and to meet tacit expectations: daunting tasks for doornails.

But robots could move farther into the social world than people currently expect, in part because that world may prove oddly welcoming. People will ascribe human feelings to, and invest their own feelings in, things which have only the most passing claim to them—cats, toys, comfort blankets. And, pace Asimov, the relationship need not always be one of fear and distrust. Especially when helped along by good design, people can be quite empathetic towards technology.

The ladder task at the DRC demonstrates the point. It was impressive to see SCHAFT reach the top of its ladder when so many others had only barely got off the ground. But it was remarkable to see Drexel University’s little Hubo reach the last step and then succumb to a gust of wind, losing its footing. The most arresting thing was not the slip itself—a safety harness stopped the robot’s fall before it could hit the ground—but the gasp of genuine dismay from the onlookers. Writers invented robots as a way of exploring human feelings about technology; the depth of those feelings may yet surprise their makers and users.