Getting computers to recognise other cars is surprisingly difficult – but super-realistic video games can help train them up

Driver 1: ready Rockstar Games

GETTING computers to recognise other cars is surprisingly difficult. Earlier this year, the first fatal autonomous car crash happened when a Tesla Model S failed to distinguish a white truck against a brightly lit sky.

Now a study has shown that self-driving cars can be taught the rules of the road by studying virtual traffic on video games such as Grand Theft Auto V (GTA V).

Although firms like Google and Uber are teaching their software by physically driving millions of miles in the real world, they also train their algorithms using pre-recorded footage of traffic. But there’s a catch: computers need hundreds of thousands of laboriously labelled images, showing where vehicles begin and end, to make them expert vehicle recognisers. That takes people a lot of time and effort.


“One evening, after a long day of hand-labelling images, I was playing Grand Theft Auto V,” says Matthew Johnson-Roberson at the University of Michigan in Ann Arbor. “I thought, ‘this is just so realistic that it would be a perfect simulation of the real world’.”

Picking out cars in a video game is a similar task to doing it in reality, with the advantage that everything comes pre-labelled because it has been generated by the game’s software. The team trained an algorithm solely using GTA V and tested it against the same algorithm trained on real-world images. The GTA-trained one performed just as well at spotting cars in a pre-labelled data set (arxiv.org/abs/1610.01983). The video-game version needed around 100 times more training images to reach the same standard – but given that 500,000 images can be generated from the game overnight, that is not a problem.

“I thought, ‘this is just so realistic that it would be a perfect simulation of the real world’“

This isn’t the first time a research group has used video games to train AI, says German Ros at the Autonomous University of Barcelona in Spain. “It’s part of a bigger movement of using simulations to train artificial intelligence, which is beginning to take off.”

Finding the right training data is difficult. “We see AI being trained on images from similar locations, at similar times of day, under similar weather conditions, and then tested under similar conditions,” says Ros. This means that it’s hard to tell whether the computers can genuinely recognise cars, or whether they have just memorised that particular data set.

Using video games can help because they often show a variety of vehicles and conditions – but the problem is still there. “There’s nothing in GTA V that looks like a city in Japan, for example,” says Ros.

We have to show that driverless cars are safer than human-driven ones. In the real world, there is a car fatality for every 100 million miles driven. Racking up that kind of distance with a prototype is not easy. “The first step could be to make sure every system has been tested using video-game-style simulations before it hits the road,” says Johnson-Roberson.

This article appeared in print under the headline “Play it for real”