Two years ago, mounting concerns about the safety of Tesla's Autopilot system were reaching a peak. In the wake of the fatal Josh Brown crash, in which a Model S drove directly into a truck at high speed while on Autopilot, serious questions were being asked about the design and marketing of a system whose hands-free operation blurred perceptions of the line between autonomous drive and driver assistance systems. Then, on the last full day of the Obama Administration, the National Highway Traffic Safety Administration (NHTSA) came out with a report [PDF] that claimed that "the Tesla vehicles crash rate dropped by almost 40 percent after Autosteer installation." With that one finding, NHTSA singlehandedly dismissed the public's concerns about Autopilot and eliminated any momentum toward regulatory action. I was immediately skeptical of NHTSA's finding since it did not seem to control for the effects of Automated Emergency Braking (AEB) and Forward Collision Warning (FCW), which the Insurance Institute for Highway Safety (IIHS) had found to reduce frontal collisions by 40%. Randy Whitfield, of company called Quality Control Systems, had questions of his own about NHTSA's Autopilot study, and submitted a FOIA request for the underlying data which was denied. Whitfield sued the agency in response, and in November of 2018 NHTSA finally released the data. Now, thanks to Whitfield's legal wranglings and subsequent analysis [PDF], we know that NHTSA's finding was based on bad data and worse analysis. The 40% crash reduction claim that Tesla has repeatedly used to defend itself, after subsequent fatal Autopilot crashes, has now been definitively debunked. This situation raises a number of questions, each more disturbing than the next, and many of which we still don't have the answers to. While we work to obtain answers to some of the broader questions, we can at least explain how NHTSA got this analysis so badly wrong. During Whitfield's FOIA lawsuit, NHTSA revealed that it had sent an information request letter to Tesla in the course of investigating the Josh Brown crash. NHTSA had asked Tesla for "The

mileage Autosteer software was installed on the vehicle," in response to which Tesla sent data documenting “Previous Mileage before Autosteer Install” and the “Next Mileage after Autosteer Install" as well as data for airbag activated crashes. This was the data that NHTSA used to obtain its 40% crash reduction finding, in a process the agency describes as “examining the sums of the miles driven prior to Autosteer activation, miles driven after Autosteer activation, airbag deployment events prior to Autosteer activation and airbag deployment events after Autosteer activation for all of the subject vehicles.”

Quality Control Systems

What Whitfield found was that only 5,714 vehicles of the 43,781 vehicles studied actually had full, verifiable data that could support NHTSA's findings. When Whitfield performed the same analysis that NHTSA had on this cohort for which complete mileage data before and after Autosteer installation was available, he found that it pointed to a 59% increase in airbag deployment after Autosteer was installed. Even then, without knowing the exact mileage at which the airbag-deploying crash took place makes it difficult to come up with specific data about the relative safety of the system. But, using statistical regression, Whitfield determined that the installation of Autosteer was not associated with a decreased risk of an airbag deployment crash, when controlling for exposure mileage. His analysis actually found that "Autosteer is actually associated with an increase in the odds ratio of airbag deployment by more than a factor of 2.4." According to Whitfield's analysis, NHTSA seems to have applied a separate calculation of exposure mileage for the 14,791 vehicles where “Previous Mileage before Autosteer Install” and the “Next Mileage after Autosteer Install” were both unreported. He shows that all the mileage for these vehicles is applied to "Next Mileage after Autosteer Install" but that three airbag deployments were counted before Autosteer was installed, fundamentally skewing the data in favor of finding a reduction in post-Autosteer installation crashes. When Whitfield looked at the 8,881 vehicles for which there was a gap between the “Previous Mileage before Autosteer Install” and the “Next Mileage after Autosteer Install” he found that the gap was 2.6 times as large as the total exposure mileage before Autosteer installation and eight percent smaller than post-Autosteer installation mileage. By comparing this gap with the cumulative mileage before and after Autosteer installation,he found evidence that there had been "a considerable undercount of actual exposure mileage before Autosteer installation" resulting in "a differential bias that inflates the calculated crash risk 'before Autosteer' to a greater degree than the calculated risk 'after Autosteer.'” When Whitfield looked at the 14,260 vehicles where the exact mileage at Autosteer installation wasn't reported, but "Next Mileage after Autosteer Install" was reported, he found that NHTSA included no exposure miles at all in the crash rate calculation even though there were 15 airbag deployments for those zero miles. Obviously counting 15 deployments over zero miles for nearly a third of the entire population studied fundamentally threw off the ultimate crash rate in a way that made it more likely to find a post-install reduction.

Quality Control Systems