The idea of traveling around town in a self-driving car has captured the world’s imagination. In addition to promising a more relaxed commute, autonomous driving is regarded as a solution to everything from traffic congestion and lack of parking spaces to accident prevention and reduced carbon emissions.

But how realistic is our vision of the car of the future?

Step-By-Step Toward Autonomous Driving

Modern cars already have a lot of assistance systems which anticipate the kinds of features self-driving vehicles will have: dashboard cameras, lane departure warning, blind spot detection, collision prevention, rain sensors, automated parking and real-time routing.

The US Department of Transport has recently announced plans to require cars to stream movement data and monitor other vehicles to avoid crashes. To achieve this, vehicle-to-vehicle communication systems will interact with other driver assistance systems such as automatic braking. They could become mandatory within four years and help reduce the 35,000 deaths a year in road traffic accidents throughout the US.

Google and Uber are among the companies which have sought a high profile for their self-driving vehicles tests, but have hit safety issues in the process. Uber recently found itself at loggerheads with cyclists in California. It emerged that Uber’s autonomous test vehicles could not deal with turning safely across bike lanes, potentially endangering cyclists. Uber spokeswoman Chelsea Kohler told the Guardian in an email that “engineers are continuing to work on the problem”, and said that the company has instructed drivers to take control when approaching right turns on a street with a bike lane.

In spite of the setbacks experienced by these pioneers, most of the world’s biggest car makers – Ford, Nissan, Honda, Daimler, Peugeot, and Hyundai, to name a few – are working on autonomous vehicles that they expect to hit our roads over the next few years. In China, Baidu also has plans for autonomous vehicles.

Tesla is among the most advanced. Tesla cars already in production come with an array of cameras, sensors, and software enabling them to operate in ‘Autopilot’ mode. However, to date, you can only do so in the right conditions – clearly marked lanes, a relatively constant speed and a map of the area you’re traveling through. That said, Tesla has announced plans to have a car self-drive from Los Angeles to New York by the end of 2017.

Supercomputers on Wheels

Business Insider estimates that by 2020, 10 million self-driving cars will be on the road. However, a lot of development still needs to happen.

Maps are key to the success of automated driving. A report in Popular Mechanics recently quoted an expert saying that while you could base self-driving entirely on visual and sensor cues, this approach would become difficult when rain or snow reduced the visibility of curbs and road markings. Therefore, the magazine concluded, there was industry consensus “that maps will make or break the age of self-driving cars.”

Compared to maps for existing satellite navigation systems, autonomous driving requires a much deeper level of information about the road network. Mapping experts at TomTom are working on highly detailed maps to address this. They will reflect a complete, three-dimensional model of the road, including the road profile, curvature, and terrain.

Self-driving cars will use Artificial Intelligence (AI) to combine data from different sources – built-in sensors, cameras, road signs, traffic and weather information – to assess their environment, learn and make decisions on what to do next. To achieve this, large amounts of data will need to be processed on board, in real time, so the car can adapt to the driving situation as needed. Chip makers including NVIDIA and Intel are focusing their research and development on providing the required processing ‘horsepower,’ which has been compared to creating a moving supercomputer.

Regulators are Stepping in

Then there is the design of these cars. When Google started its self-driving initiative – one of the first companies to do so – its proposed ‘pod’ vehicles were not going to have a steering wheel or pedals and were going to have space for just two passengers. Having recently entered into partnership with Fiat Chrysler, Google is now putting its faith into a minivan – equipped with both pedals and steering wheels.

This may also have been, at least in part, a response to new laws on the use of driverless cars. The state of California – where Google is based – has stipulated that self-driving cars must have steering wheels and brake pedals. In Michigan, regulators are working on new laws which would legalize the testing of vehicles without steering wheels and pedals – allowing for no human intervention. It is thought that Michigan wants to gain an advantage on California, which has so far been at the forefront of driverless car development.