A look at how far today’s autonomous vehicles can take us.

On the 1st of February, 2017 the California Department of Motor Vehicles released its collection of autonomous vehicle (AV) disengagement reports for reporting year 2016. Vehicle manufacturers currently developing self-driving vehicles targeted at SAE levels 4/5 are required to disclose the disengagement results of vehicles they test in full autonomous mode on public California roads. Of the 22 registered AV developers (as of February 9th, 2017), only 11 companies were required to report this year.

Disengagement reports track the number of times an AV has had to relinquish control to the driver due to either A) encountering a situation the computer could not handle or B) the driver deciding to take control. While the reports give us a bit of insight into the overall stability of each manufacturer’s system, it is not possible to directly compare a given manufacturer’s data to another.

Disengagement reports are not standardized in the information they report, and disengagements per 1000 miles — the metric most everyone focuses on — can be misleading. Consider for example a manufacturer who only tests on highways in ideal weather versus another manufacturer who tests across a broad range of weather and city/highway. The latter may have more disengagements due to more challenging situations as opposed to an inferior system.

That said, I thought it would be interesting to take each disengagement report from this year and extrapolate how far you could ride in each manufacturer’s vehicle in full autonomous mode before having the system disengage.

By computing a simple Total Miles Traveled divided by Total Disengagements, I am able to calculate an average miles traveled per disengagement (MPD). I then take each MPD and plot a route as if planning a road trip starting in the middle of San Francisco, heading north towards Seattle.

Unless otherwise noted, I only take into account miles driven in the 2016 period, i.e. December 2015 — November 2016.

Tesla (T) and Mercedes-Benz (M)

Tesla & Mercedes-Benz

Starting off with the most disengagement-prone of all the manufacturers, Tesla (3 MPD) and Mercedes-Benz (2 MPD) both keep you within San Francisco. Beginning in central SF, you could ride your autonomous Mercedes to the eastern edge of Golden Gate Park, while your Tesla would take you approximately to the entrance of Presidio. Essentially, Tesla and MB would be good enough to use for a short ride within the city, but not much else.

Tesla has never needed to report any testing it does with its publicly released Autopilot system. While Autopilot does perform automated steering and speed control, it only operates in specific scenarios and is classified as an SAE Level 2 system. As such it is exempt from disengagement reporting.

Tesla has publicly stated that its disengagement report only covers a set of vehicles that were put into full autonomous mode for marketing purposes. Per the report, only 4 vehicles were driven from October 14 to October 22, 2016. This may have been the testing that resulted in the widely publicized self-driving Tesla demonstration video.

Mercedes-Benz’ testing only consisted of 1 vehicle over a period of 6 months. Interestingly, the company decided to split up the reasons why disengagements occurred between automatic and manual. Automatic is defined by the company as the technology deciding to disengage of its own accord, whereas manual is where the driver felt uncomfortable and took control (their wording). Automatic and manual disengagements accounted for about 45% and 55% of total disengagements, respectively.

Delphi (D) and GM Cruise (C)

GM Cruise and Delphi

Now we’re at least getting out of the city, and are on our way north!

Cruise has a large advantage and takes you well into wine country (65 MPD), while Delphi’s vehicles (18 MPD) would barely get you over the Golden Gate bridge, and into the San Rafael area.

Per Cruise’s report, all of Cruise’s testing occurred in what it calls “urban driving” within the city of San Francisco. This is in contrast to Delphi’s report, which states that its driving was performed predominantly on highways and surface streets in Mountain View and Palo Alto. These are very different environments, and anyone from the Bay Area will tell you San Francisco is by far the most challenging. By factoring in the difficulty of testing in San Francisco along with Cruise’s higher MPD, I infer that Cruise’s self-driving system may be more advanced than Delphi’s.

Ford (F) and Nissan (N)

Ford and Nissan

We’re finally getting to manufacturers who have vehicles capable of something reminiscing a road trip. Both Ford (197 MPD) and Nissan (146 MPD) are capable of taking you for a ride beyond an hour long, while making sure you still get home.

Ford reported a fleet of 2 self-driving Ford Fusion Hybrids, however it only drove 1 of the Fusions for a brief period in March, 2016. Per its report, Ford only tested on I-10, a primarily straight desert highway between Los Angeles and Arizona. With such targeted and presumably ideal driving conditions, it’s no wonder Ford was able to achieve such a relatively high MPD.

In contrast, Nissan tested a total of 5 vehicles (3 Nissan Leafs and 2 Infiniti Q50s) throughout the course of 2016. The exact testing locality is not called out in the report, however testing occurred over a mix of dry city and highway driving.

BMW

Finally, we’ve made it out of California!

With a MPD of 638, BMW is the first manufacturer to take us out of the Golden State and well into Oregon — all the way to Portland in fact!

From the results, it appears BMW is much more advanced in its AV system than any other manufacturer we’ve looked at right? Tesla and Mercedes-Benz failed to leave San Francisco, while every other manufacturer is still stuck in California.

Well, not really.

Per BMW’s report, all testing only occurred from March — April and a total of 638 miles were driven with a whopping total number of disengagements = 1.

BMW’s description of the event from the report:

“There was one disengagement that occurred on April 13, 2016 on U.S. highway 101 northbound with dry roads and clear weather. The operator disengaged the automated driving system because the lane markings were not clear enough for the system to detect the lane. The vehicle alerted the driver visually. This was an unplanned disengagement with the operator’s reaction time under 2 seconds.”

A U.S. highway with clear and dry roads in the daytime is pretty much as easy of an environment to test on as one can ask for (sans traffic), and it’s interesting the disengagement was based on an inability to detect lane markings. Given this and the lack of statistically significant data (638 miles isn’t a whole lot), I am willing to venture out and say BMW’s large MPD may be due to luck and selectively choosing ideal conditions.

Waymo 2016 (W in Miami) and Waymo 2015 (Cyan marker in Montana)

Waymo

Introducing Waymo into the analysis helps put all of the aforementioned progress into perspective.

Waymo’s 2016 numbers (5128 MPD) enable us to start a road trip in San Francisco and drive through Seattle, Chicago, Boston, Washington D.C. and end in Miami before ever having to disengage the self-driving system.

Even just looking at Waymo’s report from last year, using that system we would have been able to make it all the way to Montana before experiencing a disengagement. A road trip with Waymo’s 2015 self-driving system is over double the distance than a road trip with the next best 2016 self-driving system.

In truth, these are the results I would expect from Waymo, however visualizing the progress of the technology company in terms of a 78-hour road trip across the United States of America really drills the point home.

How many people in your personal network do you trust to drive you non-stop on a trip of that size?