Don’t get too cocky now Jason Lee/Reuters

Confidence in your abilities is usually a good thing – as long as you can recognise when it’s time to ask for help. As we build ever smarter software, we may want to apply the same thinking to machines. An experiment that explores a robot’s sense of its own usefulness could help guide how future artificial intelligences are built.

Overconfident AI can cause all kinds of problems, says Dylan Hadfield-Menell at the University of California, Berkeley. Take Facebook’s newsfeed algorithms, for example. These are designed to suggest articles and posts that people want to see and share. Yet by following this remit unquestioningly, they have ended up filling some people’s feeds with fake news.

For Hadfield-Menell and his colleagues, the answer is to make AIs that seek and accept human oversight. “If Facebook had this thinking, we might not have had such a problem with fake news,” he says.


Rather than pushing every article it thinks Facebook users want to see, an algorithm that was more uncertain of its abilities would be more likely to defer to a human’s better judgement.

Off-switch game

The Berkeley team designed a mathematical model of an interaction between humans and robots called the “off-switch game” to explore the idea of a computer’s “self-confidence”.

In this theoretical game, a robot with an off switch is given a task to do. A human is then free to press the robot’s off switch whenever they like, but the robot can choose to disable its switch so the person cannot turn it off.

Robots given a high degree of “confidence” that what they were doing was useful would never let the human turn it off, because they tried to maximise the time spent doing the task. In contrast, a robot with low confidence would always let a human switch it off, even if it was doing a good job.

Yet Hadfield-Menell does not think we should make an AI too insecure. If an autonomous car was driving a young child to school, the car should never allow the kid to take control, for example. In this case, the AI should be confident that its own abilities outstrip the child’s, whatever the situation, and refuse to let the child turn it off.

The safest robots will strike a balance between the two extremes, says Hadfield-Menell.

Vital framework

AIs that refuse to let humans turn them off might sound far-fetched, but such considerations should be critical for anyone making robots that work alongside humans, says Marta Kwiatkowska at the University of Oxford.

Machines such as driverless cars and robot firefighters will be asked to make decisions about human safety, so it is vital that the ethical framework for these decisions is put in place sooner rather than later, she says.

The off-switch game is only the start, says Hadfield-Menell. He plans to explore how a robot’s decision-making changes when it has access to more information about its own usefulness. For example, a coffee-making robot might consider its task more useful in the morning.

Ultimately, he hopes his research will lead to AI that is more predictable and makes decisions that are easier for humans to understand. “If you’re sending a robot out into the real world, you want to have a pretty good idea of what it’s doing,” he says.

Reference: arxiv.org/abs/1611.08219v2

Read more: Give your car a conscience: Why driverless cars need morals