Using a head-up display and a cap that reads brain activity, for the first time three people with spinal injury have controlled a robot and seen what it sees

Capable of hosting us Hironori Miyata/Camera Press

IN THE 2009 Bruce Willis movie Surrogates, people live their lives by embodying themselves as robots. They meet people, go to work, even fall in love, all without leaving the comfort of their own home. Now, for the first time, three people with severe spinal injuries have taken the first steps towards that vision by controlling a robot thousands of kilometres away, using thought alone.

The idea is that people with spinal injuries will be able to use robot bodies to interact with the world. It is part of the European Union-backed VERE project, which aims to dissolve the boundary between the human body and a surrogate, giving people the illusion that their surrogate is in fact their own body.

In 2012, an international team went some way to achieving this by taking fMRI scans of the brains of volunteers while they thought about moving their hands or legs. The scanner measured changes in blood flow to the brain area responsible for such thoughts. An algorithm then passed these on as instructions to a robot.


“The feeling of embodying the robot was good, although the sensation varied over time“

The volunteers could see what the robot was looking at via a head-mounted display. When they thought about moving their left or right hand, the robot moved 30 degrees to the left or right. Imagining moving their legs made the robot walk forward.

Now, a second team has tested a similar set-up in people who are paralysed from the neck or trunk down. To make the technology cheaper, more comfortable and more portable, the team swapped the fMRI scanner for an electroencephalogram (EEG) cap, which records electrical activity in the brain using electrodes attached to the scalp.

Each of the three volunteers in Italy donned the cap plus a head-mounted display that showed what a robot – in a lab in Tsukuba, Japan – was looking at. To move the robot, they had to concentrate on arrows superimposed across the display, each flashing at a different frequency. A computer could detect which arrow a participant was staring at using the EEG readings that each frequency provoked. It then sent the corresponding movement to the robot.

The set-up allowed the volunteers to control the robot in near real time. They were able to pick up a drink, move across the room and put the drink on a table. “It took just 6 minutes of training to start using the technology,’ says Emmanuele Tidoni at the University of Rome. “The feeling of actually embodying the robot was good, although needless to say, the sensation varied over time,” said Alessandro, one of the volunteers living with spinal cord injury. “When the robot was stationary the feeling of embodiment was low, but the moment I gave the first command or changed direction, there was this feeling of control and increased embodiment.”

The team also tried to boost this feeling using auditory feedback. While controlling the robot, both able-bodied volunteers and people with spinal cord injury managed to place a bottle closer to a target location when they heard footsteps as they walked, rather than a beep. The improved control suggests they feel more in tune with the robot itself, says Tidoni.

The project also studied how this technology could be used in social interactions, by getting people in Italy to play a chess-like game via the robot against an opponent located in Munich, Germany. The results will be published later this year.

Alessandro is excited about the potential. “It’s a sensitive and important issue, but will certainly have a major impact on the way we all can communicate to each other,” he says.

But he hopes that the implications for mental health will also be looked at. “What will happen to a person who cannot move in real life after they use this technology intensively? Will they still feel isolated or lonely?” he asks. “Any developments also need to study the impact that these technologies may have on the psychological well-being of people with various degrees of disability.”

Although we’re not yet at Surrogates-level immersion, the technology could one day dramatically improve the lives of people with paralysis, says Noel Sharkey at the University of Sheffield, UK. “This is a very long way off, but getting towards that for people who otherwise can’t move would be astounding.”

This article appeared in print under the headline “Being in a robot’s shoes”