A comparison of Lidar(left), Structure From Motion(Middle), and Sparse Space Carving(Right) used in autonomous car research.

As you can see, there are caveats to using a feature based system, and limited camera coverage limits how far a single car can process. Feature based techniques have errors when there are no textures, or too many similar textures. Now let’s lay out some new premises, then logically follow them to a better solution:

Autonomous cars can accurately model their surroundings(not true right now, but there is a better solution). Using accurate data of surroundings, a completely algorithmic system will always be safer than a mix with humans and autonomy. Thus, autonomous driving could be guaranteed safer, only if all cars are autonomous. Otherwise you still risk getting hit by human drivers. If an autonomous car is more prone to be hit by human drivers, then it’s possibly not safer than regular driving in all situations. Cities are where autonomous cars have been hit the most by human drivers. Snow, rain, occlusions, and weak road markers can limit the amount of accurate data autonomous cars independently gather. Tesla warns the driver if the the car can’t see enough to navigate.

Given these premises, it follows that a solution would be safer if all cars were autonomous, with completely accurate models of surroundings. So the important question is..

How can we quickly make all cars autonomous and safer ?

The solution is to disconnect our 3D modeling from the cars themselves, and instead build cities to accurately generate dense models from fixed cameras and drones.

This proposed system can be maintained and upgraded faster than cars, and in total it would cost less than buying millions more cars. This solution allows generating much more accurate models than you can from the tiny computer you can cram into cars. Once you enter an upgraded city, you would switch to autopilot, and be guaranteed that driving is as safe as possible within the city.

Traffic control would be more efficient, because a single dense model can see problems far down the road, and re-route everyone smoothly. Since the 3D model is de-coupled from the car, it doesn’t matter where the lines on the road are. The cameras in my automated platform self-calibrate their positions, and thus would always know where the car is relative to them and the road. This doesn’t work if the sensors are attached to the car, when the car itself can’t see the surroundings. Using sparse infrared cameras, you can see through snow and rain. Drone or roadside infrared cameras can be shielded from above, or fly above the clouds, and still see through weather.

We can upgrade one city at a time, and guarantee all driving is as safe as possible in those cities, by upgrading cars to listen, and give up control to the city. You don’t have to buy a new autonomous Tesla, but if you do, you can be sure that there won’t be human drivers hitting you in these cities. Once cities are safe, we can deploy drones and road side cameras to control highways and rural areas.

Since the model would always be on, it could see far down the road. So the probability of that nightmare situation of rushing up on a large group of pedestrians is minimized. A city wide platform-based solution can support a hybrid between self-driving and autonomy, meaning it allows you to drive until you approach another person, and then takes over to guarantee collisions are impossible.

Check out the video below for a simulation which tests this reasoning. To test this theory, I simulated a city as input to a real time 3D reconstruction algorithm, and used the output models to drive a traffic control system. The cars are all initialized with different random speeds, so they would definitely crash without accurate controls.

Atlas Autonomous Car Test Framework. 3D Reconstruction on the left is used to drive the traffic control algorithm on the right.

The simulated cameras, including a moving drone, are used as input to the reconstruction algorithm running in the bottom left corner. The algorithm renders a dense 3D point cloud, which it maps to a 2D traffic control grid. The grid is used to control every car through the city safely. With autonomous cars, you can turn the traffic lights off, and increase efficiency, but it looks more disorganized.

This platform was originally built to reconstruct dense models in real time from arbitrarily moving cameras for live sports. You can move cameras, and they will self calibrate to render a live moving mesh, with a 100-500 millisecond delay from live action to devices. You can see how it renders people in real time here:

It turns out the technology is useful for autonomous cars today, so I’m releasing my research. Using this platform we can use fewer cameras to rig a city with a single, dense, real time 3D reconstruction to build a city wide traffic control system.

What’s the catch? Honestly, I believe this to be the best solution forward, that will save the most lives. If I had to nitpick, I would use the following arguments:

The connection from city to car could be interrupted.

True. However, since only control data is sent to the cars, this is more efficient than car based systems as it allows a more accurate model to run on many servers. It relies on a good wireless connection. The intermediate solution until cities upgrade can be like Tesla’s: the car warns the driver if they need to put the hands on the wheel. In the long term, cities will have to invest more in local signaling solutions to ensure a robust network for cars. This won’t sell more cars.

True. However this solution can save more lives sooner. There’s a great short story on folklore.org about Steve Jobs motivating the old Macintosh team to cut the boot time by 10 seconds. His reasoning was that if there are millions of Macs, cutting 10 seconds would add up to many lifetimes over a year. So by doing this for him, they would save many lives. Engineers will want to save lives over selling more cars, so I’m not worried about this argument. This technology can actually save millions of lives directly. The WHO recorded 1.24 Million traffic deaths in 2010. For companies like google, it’s arguable that it’s in their best interest to save millions of lives, since those millions would be driving future google ad revenue. Google may license technology to car manufacturers, but the business from ad revenue by saving lives is increased regardless of what car people buy. According to Forbes estimates, google makes 10.09 per user per year. By saving 1.24 Million lives, google could have earned ~12.5 million or more per year along their lifetime. What about Hacking?

If you’re on your own with no connection to a central server, no one with expertise can help you. If everyone if using the same dense network of 3D reconstructions from central servers, hacking will be easier to detect and guard against. The traffic control algorithms can also be upgraded faster if it’s all in one place, allowing fixing security holes much faster than updating each car. Car manufacturers wouldn’t have to recall millions of cars if there’s a new hack. So it turns out all those hackers, that are finding ways to control a car, are actually helping to build the knowledge we need to upgrade older cars. Their hacks also point out the security features we would need in such a system, before we build it. Is this more expensive?

No, this is less expensive in the long run for all parties except car sellers. According to an NHTSA study, the societal costs related to crashes for the US annually is $826 billion. The New York Post claims NYC loses $4B per year to crash related expenses. Consumers can get a guaranteed safer drive without buying a new $40,000 car. Recalling cars is also orders of magnitude more expensive than updating the control firmware in each car. Cities can pay to upgrade the cars of their citizens, and still have more left over from the savings in avoided crashes.

Congress is mulling a bill that seems like a knee jerk reaction to these assumptions. If you like this solution, then the car hackers are actually paving the way for ubiquitous autonomous cars. A draft to ban car hacking is linked here: http://docs.house.gov/meetings/IF/IF17/20151021/104070/BILLS-114pih-DiscussionDraftonVehicleandRoadwaySafety.pdf

I will continue to improve the simulation, such as by modeling real cities, snow, pedestrians, different road types, and human drivers. The reconstruction engine is also available to any city which wants to test real cameras as input to their own reconstruction platform.

So how can you help? Share this reasoning with the engineers working to solve these problems. If you have ideas to improve the simulation, feel free to comment or reach me at: rohanarun@gmail.com. My startup Reconstruction Engine(www.reconstructionengine.com) is working to solve the hard problems in real time 3D reconstruction.

Sources:

1) http://www.forbes.com/sites/tristanlouis/2013/08/31/how-much-is-a-user-worth/

2) http://www.nhtsa.gov/About+NHTSA/Press+Releases/2014/NHTSA-study-shows-vehicle-crashes-have-$836-billion-impact-on-U.S.-economy,-society

3) http://nypost.com/2015/03/20/car-accidents-cost-nyc-nearly-4-billion-a-year/

4) http://www.who.int/violence_injury_prevention/road_safety_status/2013/en/

5) http://www.folklore.org/StoryView.py?story=Saving_Lives.txt

Investors: https://angel.co/reconstruction-engine-1