Every year, companies that operate self-driving cars in California are required to submit data to the state’s Department of Motor Vehicles listing the number of miles driven and the frequency at which human safety drivers were forced to take control of their autonomous vehicles (also known as a “disengagement”). And every year, those same companies raise a huge stink about it.

Waymo, which drove 1.45 million miles in California in 2019 and logged a disengagement rate of 0.076 per 1,000 self-driven miles, says the metric “does not provide relevant insights” into its technology. Cruise, which drove 831,040 miles last year and reported a disengagement rate of 0.082, says the “idea that disengagements give a meaningful signal about whether an [autonomous vehicle] is ready for commercial deployment is a myth.” Aurora, which only drove 13,429 miles and recorded a disengagement rate of 10.6 per 1,000 miles, calls them “misguided.”

Media outlets typically lunge for the disengagement reports as the only scrap of transparency on robotaxi progress they tend to get

Autonomous vehicle (AV) companies can be a black box, with most firms keeping a tight lid on measurable metrics and only demonstrating their technology under the most controlled settings. That means the disengagement reports can feel like the only scrap of transparency on AV progress most of us get. The problems generally stem from trying to use the reports in ways they aren’t designed to be used: to determine who’s driven the most miles, who has the best disengagement rate, and who is generally furthest along in developing autonomous vehicles.

“Comparing disengagement rates between companies is worse than meaningless: It creates perverse incentives,” said Bryant Walker Smith, associate professor at the University of South Carolina’s School of Law and an expert in self-driving cars. For instance, Smith says, if he were to register in California and never test, he’d look good. “If I wanted to look even better, I’d do a ton of easy freeway miles in California and do my real testing anywhere else,” he added.

Disengagement reports aren’t the best way to build trust and credibility in autonomous vehicles, Smith points out. Instead, releasing testing summaries with detail and context is better. But no company to date has done that.

“If I just wanted easy legitimacy for prospective investors, I’d register in California and never test.”

Most of the major players have submitted voluntary safety reports to the federal government as part of the Department of Transportation’s voluntary guidance. Unfortunately, these safety reports usually are little more than slick marketing materials.

As for the disengagement reports, they’re “effectively meaningless,” said Sam Abuelsamid, senior analyst at Navigant. There are two big problems: companies have a lot of discretion about when to disengage, and the testing environments aren’t uniform. Inevitably, that means it’s impossible to make an apples-to-apples comparison between companies. Abuelsamid would rather see a standardized performance evaluation of the systems before anyone gets a permit for testing on the streets.

The DMV notes that though the reports are a requirement for permit holders, they should not be used to “compare one company with another or reach broad conclusions on technological capabilities,” said the agency’s public information officer, Martin Greenstein.

“Permit holders all have different goals.”

“Permit holders all have different goals and business models, and are testing in different ways, locations and conditions with different amounts of vehicles,” Greenstein said in an email. He added that companies aren’t required to report testing on private roads or test tracks, and they don’t have to disclose testing that occurs out of state. Testing of advanced driver assist systems, like Tesla’s Autopilot, or testing done in simulation is also not required.

It’s a fresh reminder that most of what we know about public testing of self-driving cars comes from the companies themselves — which raises a lot of questions. There are almost no requirements for companies doing public testing in any other state. California’s disengagement reports are the real outlier. And even those offer what, at best, can be described as an incomplete picture and, at worst, is a misleading one of the progress of the technology.

So with that in mind, here are the latest numbers from some of the major players.

Waymo

Waymo, the self-driving unit of Alphabet, has 153 self-driving vehicles and 268 drivers permitted in the state of California. Most of its testing in California is around its Mountain View campus. (The bulk of Waymo’s fleet is located in the Phoenix area where it has a limited robotaxi operation serving around 1,500 people. That’s an estimated 600 self-driving vehicles; the number of drivers is unknown.)

The company says it drove 1.45 million miles in California in 2019, a slight increase over the company’s 1.2 million miles in 2018. The company’s disengagement rate dropped to 0.076 per 1,000 self-driven miles or one disengagement per 13,219 miles.

Cruise

Cruise, a majority-owned subsidiary of General Motors, has been ramping up its presence in San Francisco after failing to meet its own deadline of launching a full-scale robot taxi service by the end of 2019. The company now has 233 autonomous vehicles permitted to test on public roads, the majority of which operate in the Bay Area, according to the DMV.

Cruise drove 831,040 miles in 2019, almost doubling the number of miles it drove the previous year. It recorded a total of 68 disengagements — 43 of which were in the first six months. That means Cruise’s disengagement rate is 0.082 per 1,000 miles driven, or one disengagement per 12,221 miles driven.

Aurora

Aurora, the self-driving car company founded by former Waymo engineer Chris Urmson, has only nine cars registered in California. Urmson founded the company in 2017, along with Sterling Anderson, Tesla’s former head of Autopilot, and Drew Bagnell, a founding member and head of Uber’s autonomy and perception team. As such, it’s still very much early days for the company.

Aurora says it drove 39,729 miles on public roads, of which 26,300 were driven by people. While testing in autonomous mode, the company says it experienced 142 disengagements, a quarter of which were caused by “a single software issue” that was addressed earlier in the year. Its rate of disengagements per 1,000 miles was 10.6, which is similar to last year.

Nuro

Nuro is another company founded by ex-Waymo engineers. The company is trying to launch a delivery service Texas using its purpose-built, egg-shaped R2 autonomous vehicles. But it is still doing testing in California where it is based. Nuro has 36 vehicles registered in the state.

The company says it drove 68,762 miles in autonomous mode in 2019 and logged 34 disengagements. Nuro’s disengagement rate is 0.49 per 1,000 miles, or one every 2,022 miles driven.

As of February 5th, there are 66 permit holders and 769 autonomous test vehicles registered with the California DMV, though it’s difficult to gauge the average number that can be found on the road at any given time. The agency said the combined field drove over 2.9 million miles in 2019.

Other major permit holders include Tesla (with 32 vehicles registered with the DMV), Zoox (58 vehicles), Apple (66 vehicles), Pony.ai (22 vehicles), Lyft (20 vehicles), Mercedes-Benz (19 vehicles), Toyota Research Institute (14 vehicles), Argo.ai (14 vehicles), Nio (13 vehicles), Didi Research (12 vehicles), and Nvidia (11 vehicles).

Here are some more numbers: