Next time you’re driving down the road or walking down the street, pause to consider how you read your surroundings. How you pay extra attention to the kid kicking a soccer ball around her front lawn and the slightly wobbly, nervous looking cyclist. How you deprioritize the woman striding toward the street, knowing she’s heading for the group of friends waving to her from the sidewalk.

You make these calls by drawing on a lifetime of social and cultural experience so ingrained you hardly need to think about it. But imagine you’re an autonomous car trying to do the same thing, without that accumulated knowledge or the shared humanity that lets you read others’ nuanced behavioral cues. Treating every pedestrian, cyclist, and vehicle as an obstacle to be avoided might keep you from hitting anything, but it could just as easily keep you from getting anywhere.

“We call it the freezing robot problem,” says Anca Dragan, who studies autonomy in UC Berkeley’s electric engineering and computer sciences department. “Anything the car could do is too risky, because there is some worst-case human action that would lead to a collision.”

Expect a thaw. Researchers like Dragan are tackling the challenges of interpreting—and predicting—human behavior to make self-driving cars safer and more efficient, but also more assertive. After all, if every machine screeches to a stop for every unpredictable human, we’ll have soon millions of terrified robots choking the streets.

To prevent the clog, those researchers are leaning on artificial intelligence and the ability to teach driving systems, through modeling and repetitive observation, what behaviors mean what, and how the system should react to them.

TU Delft

That begins with recognizing that people are not, in fact, obstacles. “Unlike, say, a tumbleweed moving along the street under the wind's effect, people move because they make decisions,” Dragan says. “They want to do something, and they act to achieve it. We’re first looking into inferring what people want based on the actions they've been taking so far. So their actions are rational when seen from [that perspective], and would appear irrational when seen from the perspective of other intentions.”

Say a driver in the right lane of the freeway accelerates. The computer knows people should slow down as they approach exits, and can infer this person is likely to continue straight ahead instead of taking that upcoming off ramp. It’s a basic example that makes the point: Once computers can estimate what humans want and how they might achieve it, they can reasonably predict what they’ll do next, and react accordingly.

Machines en Scene

The key, even with machine learning, is to look beyond the individual elements of a scene. “It’s important to make strides there, but it’s only seeing part of what’s going on in a roadway setting,” says Melissa Cefkin, a design anthropologist at Nissan’s Silicon Valley R&D center. “We’re really good as human beings at recognizing certain kinds of behaviors that look one way to a machine, but in our social lens, it’s something else.”