THE MODERN AUTOMOTIVE era began with a competition. In the early 1890s there was much interest in the emerging technology of horseless carriages, which promised to combine the speed of a train with the flexibility of a horse and the convenience of a bicycle. Le Petit Journal, a French newspaper with a knack for publicity stunts, decided to hold a contest to discover the best method of propulsion: steam, electricity or petrol engine. It invited entrants to drive from Paris to Rouen, a distance of 79 miles. Their vehicles would be judged not by their speed but whether they were safe, easy to use and economical to run.

The competition, held in July 1894, attracted crowds of onlookers as 21 contraptions set out from Paris. Only 17 vehicles stayed the course; along the way, seven dogs were run over and one cyclist was injured. The clear winner was not a direct participant but an inventor: Gottlieb Daimler, whose internal-combustion engine had powered nine of the vehicles, including the four that shared first prize. He had, the judges proclaimed, “turned petroleum or gasoline fuel into a practical solution” for self-propelled vehicles, which were starting to be referred to in French as “automobiles”. Daimler’s victory helped establish the supremacy of petrol-powered cars in the 20th century, and the term automobile soon spread into English and other languages.

Fittingly, the modern era of autonomous vehicles also began with a competition, held in March 2004 in the Mojave desert. It was organised by DARPA, America’s main military-research agency, and required driverless vehicles to navigate a 150-mile off-road course. A total of 21 teams qualified, but after pre-contest trials and mishaps only 12 vehicles crossed the starting line. Amid mechanical failures and encounters with ditches, none of them made it to the finish. Carnegie Mellon’s Sandstorm, the vehicle that did best, travelled 7.4 miles before getting stuck; as it tried to free itself, its front wheels caught fire.

It seemed that DARPA had set the bar too high. Yet when it held another competition in October 2005, five of the 23 participants completed the entire 132-mile course, and all but one beat the 7.4-mile record from the previous year. The winning vehicle was built by a team from Stanford University led by Sebastian Thrun; Sandstorm finished second. In just 18 months, autonomous driving had gone from hopeless to feasible. In a third DARPA contest, in November 2007, vehicles had to complete tasks in a simulated urban environment, coping with road signs, traffic signals and other vehicles. Six out of 11 teams completed this much more complex challenge.

Encouraged by this rapid progress, Google set up a self-driving car project in 2009, led by Mr Thrun. Since then the participants in the various DARPA contests have gone on to work on autonomous-vehicle technology at Google, Uber, Tesla and a host of startups. Prototype self-driving cars first took to America’s public roads in 2012; they have since travelled millions of miles and have become steadily more capable. But the technology is not ready for mass deployment just yet. A fully autonomous car must solve three separate tasks: perception (figuring out what is going on in the world), prediction (determining what will happen next) and driving policy (taking the appropriate action). The last of these is the simplest, making up just 10% of the problem, says Mr Thrun; perception and prediction are the hard parts.

I see

Autonomous cars perceive the world through a combination of sensors including cameras, radar and LIDAR—a radar-like technique that uses invisible pulses of light to create a high-resolution 3D map of the surrounding area. The three complement each other. Cameras are cheap and can see road markings, but cannot measure distance; radar can measure distance and velocity, but cannot see in fine detail; LIDAR provides fine detail but is expensive and gets confused by snow. Most people working on autonomous vehicles believe a combination of sensors is needed to ensure safety and reliability. (Tesla is a notable exception: it hopes to achieve full autonomy without the use of LIDAR.) High-end LIDAR systems currently cost tens of thousands of dollars, but startups are devising new solid-state designs that should eventually reduce the price to a few hundred dollars.

Having combined the data from its sensors, the car needs to identify the items around it: other vehicles, pedestrians, cyclists, road markings, road signs and so forth. Humans are much better at this than machines, which have to be trained with lots of carefully labelled examples. One way to obtain them is to pay people to label images manually. Mighty AI, based in Seattle, has an online community of 300,000 people who carefully label images of street scenes for a range of automotive clients. “We want cars to have human judgment,” says Mighty AI’s boss, Daryn Nakhuda, “and for that we need human expertise.” Imagery from video games such as “Grand Theft Auto”, which features strikingly realistic street scenes, can also help. Because the game software knows what everything is, it can label such scenes with perfect accuracy, allowing them to be used for training.

The hardest things to identify, says Mr Thrun, are rarely seen items such as debris on the road or plastic bags blowing across a highway. In the early days of Google’s AV project, he recalls, “our perception module could not distinguish a plastic bag from a flying child.” Puddles on the road also caused confusion. Combining data from multiple sensors, however, can reveal whether an item in the road is a solid obstacle or not. Cars can also compare their sensor readings with those gathered previously by other cars on the same road, learning from each other’s experiences in a process called “fleet learning”. That gives an edge to first movers with thousands or millions of miles of self-driving experience under their belts; but some startups also create and sell ready-made high-resolution maps for use by AVs.

Once a vehicle has identified everything around it, it needs to predict what will happen in the next few seconds and decide how to respond. Road signs, traffic lights, brake lights and turning signals provide some clues. But AVs are at a disadvantage to human drivers, who are used to dealing with exceptions to the normal flow of traffic, such as roadworks, broken-down vehicles, delivery trucks, emergency vehicles, fallen trees or bad weather. Snow is a particular challenge: LIDAR systems must be carefully tuned to ignore falling snow, and accumulations of snow on the roads make high-resolution street maps less accurate.

While the technology is still being developed, it helps to stick to limited areas that have been mapped in detail and generally have good weather. That explains why Phoenix, with its sunshine and regular grid system, is a popular place to test AVs. Pittsburgh is considered more difficult because of its harsher weather. Cruise, an AV startup now owned by GM, has demonstrated some impressive autonomous driving in the complex streets of downtown San Francisco. Kyle Vogt, Cruise’s boss, argues that testing in densely populated environments means cars experience unusual situations more often, and thus learn faster.

When an AV gets confused and does not know how to respond, or makes the wrong decision, the safety engineer in the driving seat takes over. This is known as a “disengagement”, and the number of disengagements per 1,000 miles travelled provides a crude measure of how the companies developing AVs compare (see chart). Disengagements are best seen not as failures but as learning experiences that help AV systems improve. Sensor data recorded in the lead-up to a disengagement can reveal what the car got wrong, says Noah Zych, head of safety at Uber’s AV unit. Modifications to its software can then be tested in simulation. “We can play it back again and again, vary the scenario and see the distribution of outcomes,” says Mr Zych. The improved software is then rolled out in real cars.

What do I do now?

Even when AVs are widely deployed, they will probably still need to ask for human assistance sometimes. Consider an AV caught behind a broken-down truck on a two-lane road with a solid line down the middle, says Christophe Sapet of Navya, a maker of driverless shuttles. Because it has been programmed to obey road markings, the AV will get stuck. Human drivers would simply bend the rules and drive around the truck when the road is clear. Navya’s AVs instead call a remote supervision centre, where a human operator can see live feeds from their cameras. Rather than controlling such a vehicle remotely, the operator gives it temporary permission to cross the white line when it is safe to do so. Mr Thrun suggests such operators could, in future, end up supervising thousands of AVs at a time.

Meanwhile, limited forms of autonomy are being gradually added to existing production cars. A scale devised by the Society of Automotive Engineers defines the levels of autonomy. Level 1 involves basic assistance (such as cruise control). Level 2 adds features such as lane-keeping, allowing the car to drive itself on highways, but still requiring the driver to pay full attention. The new Audi A8, launched this year, is the first car to achieve level 3, which means it can drive itself and monitor its surroundings, but the driver must take back control when asked.

Waymo, Uber and others are attempting to jump directly to level 4, which provides full autonomy under certain conditions, such as in a specific part of a city. Some in the industry consider the partial automation of levels 2 and 3 to be unsafe, because drivers are still required to pay attention even when they have handed over control of the vehicle, which they find hard to do. (The driver of a level 2 Tesla Model S was killed when his vehicle hit a lorry in May 2016; investigators found that despite warnings from the car, he failed to keep an eye on the road.)

One problem for AVs is that the world was built to cater for human drivers, with whom they must share the roads. Humans communicate by flashing their lights and using other non-verbal cues, which (like other driving customs) can vary from place to place. AVs will probably end up being tuned to fit in with their surroundings, driving more aggressively in Boston than in California, for example, suggests Amnon Shashua, the boss of Mobileye, a maker of autonomous-driving technology.

The public seems concerned mainly about two potential risks associated with AVs: ethical dilemmas and cyber-attacks

“You have to make the vehicle so it can operate in the world as it is today,” says Chris Urmson of Aurora, an autonomous-driving startup. But things should get easier in future. There may be road lanes or entire districts dedicated to AVs, and special equipment to support them, known as vehicle-to-infrastructure (V2I) technology. Already, in some areas where AVs operate, traffic lights have been modified to tell approaching vehicles when they will change. In future, V2I and V2V (vehicle-to-vehicle) technology should allow AVs to co-ordinate their actions better.

The public seems concerned mainly about two potential risks associated with AVs. The first is how they should respond to ethical dilemmas: say, choosing between hitting a group of children in the road or swerving and hitting another vehicle. Many people working in the field think that such questions do not reflect the real world, and point out that the best course of action is usually to slam on the brakes. AVs have superhuman, 360-degree perception and much faster reaction times, notes Danny Shapiro of NVIDIA, a chipmaker whose products power AVs.

The second worry is about cyber-attacks. AVs, which are essentially computers on wheels, could be remotely hijacked or sabotaged. Engineers working on AVs insist that they take cyber-security very seriously, and say that the multiple redundant sensor and control systems they build in to make a vehicle mechanically safe will also provide some protection. If any part of the vehicle starts to behave strangely, for whatever reason, it will stop. “It is easier to use an ordinary vehicle to kill people than to take control of a driverless car,” says Mr Sapet.

AVs are on the cusp of working on public roads, at least in orderly environments with good weather. “Once you can crack that nut, it’s incremental,” says Mr Urmson. For his part, Mr Thrun has moved on to a new challenge: building cars that fly. Automotive bosses think he is crazy, he admits. But until quite recently they were just as sceptical about self-driving cars.