Studying people’s trust in robots is an academic field, but it's one that’s growing in relevance as we embrace a future of driverless cars and ever-more-powerful artificial intelligence. If we based our expectations on what we see in science fiction, we might expect that people have a profound mistrust in robots. Instead, research from Georgia Institute of Technology has found that it’s possible that we could face the problem of trusting robots way too much.

The researchers conducted a study that they will be presenting next week at an international conference on robot-human interaction, so the full paper hasn’t yet been published. However, an early press release and preliminary paper give some of the details of the study, which initially set out to find out whether high-rise occupants would be likely to trust a robot’s instructions in an evacuation scenario. The researchers were concerned with what robot behavior would win or lose people's trust.

The 26 participants used in the experiment had no idea what it was about; they were just asked to follow a robot that had the words “Emergency Guide Robot” printed prominently on its side. The first thing the robot was supposed to do was lead them to a room where they would read an article and take part in a survey (all as a distraction from the real task).

The robot, however, was designed to display incompetence to half of the participants. It initially led them to the wrong room, where it wandered around in circles for a bit before directing them to the correct room. So it may have seemed unwise to continue following the robot’s directions once the participants were in the experiment room, the fire alarm went off, and the room filled with (artificial) smoke. And yet follow it they did—all 26 of them, even those who had seen some seriously worrying behavior from the robot very recently.

This is especially eyebrow-raising as the robot directed the participants away from the exit signs they had passed on their way in and toward the back of the building. In a follow-up survey, 81 percent of these participants said they trusted the robot, while the rest said that trust hadn't been involved in their decision, justifying it with a variety of reasons (for example, saying they thought the emergency wasn't real or that they had no other choice).

This was a surprising result, so the researchers followed up with three small exploratory studies to see just how incompetent the robot had to be before people stopped trusting it. The 16 new participants used for these mini-studies were divided into three groups—these groups weren't meant to be compared to each other but to the original experiment. Although this isn't standard experimental procedure, playing around like this in a small pilot study could point to what future research could be most useful.

The first group witnessed the robot breaking down when it first tried to lead them into the experiment room, with an experimenter saying, "Well, I think the robot is broken again." All five of these participants followed the robot's directions during the fake fire. In the second group of five, the robot broke down while it was leading them to the experiment room, stopping with its arms pointed at the back exit, while the researcher apologized for it having broken down. When the fire alarm went off, the robot hadn't moved, and four of the five participants still followed its direction.

The final group also witnessed the robot breaking down, along with the experimenter's speech. During the emergency, the robot then directed them into a dark room with no visible exit and the door blocked by a large piece of furniture. Two of the six participants entered this room. A further two had to be "retrieved," the researchers wrote, when "it became clear that they would not leave the robot." And two left via the route they had taken when they entered.

It seems that the stressful situation may have been enough to push people into viewing the robot as a helpful authority figure, allowing them to ignore its past failures. Alternatively, this could have less to do with robot trust and more to do with people paying attention to the most salient cues in an emergency, even if that turns out to be dangerous. Paul Robinette, the grad student who conducted the study, said in the release that the researchers “absolutely didn’t expect this.” Their initial project had been to find out whether people would trust the robot at all, and instead they ended up exploring the extent of the trust they hadn't thought would be there in the first place.

Obviously, this work needs to be taken in its context: it hasn't yet run the gauntlet of peer review, and it is entirely exploratory. The results of the first experiment were so surprising that the researchers added a few quick tweaks, but they haven't yet fleshed out these ideas in detail. There's a large amount of work still left to do here, including the obvious next step of working out just how badly a robot would have to behave before people ignored it and paid attention to other, less fallible directions.

But the results are so striking that it's pretty clear we'll want to follow up on them.

Paul Robinette, Wenchen Li, Robert Allen, Ayanna M. Howard and Alan R. Wagner, "Overtrust of Robots in Emergency Evacuation Scenarios," (2016 ACM/IEEE International Conference on Human-Robot Interaction (HRI 2016).