Researchers from Bochum, Germany, and San Diego, California, say they’ve found the precise mechanisms that allowed diesel Volkswagens and Audis to engage or disengage emissions controls depending on whether the cars were being driven in a lab or driven under real-world conditions. As a bonus, the researchers also found previously-undisclosed code on a diesel Fiat 500 sold in Europe.

Auto manufacturers have been cheating on emissions control tests for decades, but until recently, their cheats were fairly simple. Temperature-sensing or time-delay switches could cut the emissions control system when a car was being driven under certain conditions.

These days, cars are an order of magnitude more complex, making it easier for manufacturers to hide cheats among the 100 million lines of code that make up a modern, premium-class vehicle.

In 2015, regulators realized that diesel Volkswagens and Audis were emitting several times the legal limit of nitrogen oxides (NO x ) during real-world driving tests. But one problem regulators confronted was that they couldn’t point to specific code that allowed the cars to do this. They could prove the symptom (high emissions on the road), but they didn’t have concrete evidence of the cause (code that circumvented US and EU standards).

Luckily, subsequent subpoenas of e-mails between executives revealed the kind of broad-brush information that helped federal investigators secure settlements from Volkswagen and its supplier Bosch. Investigators were also able to get a plea deal with a former VW engineer.

This latest research finally offers a smoking gun. For more than a year, researchers studied 926 firmware images from the VWs and Audis identified by the EPA in 2015, and they found a potential defeat device in 406 of those firmware images. All the cars studied had Engine Control Unit (ECU) systems developed by Bosch.

Interestingly, Volkswagen may not have written any of the code that enabled its scandal, although it may have requested certain functions from Bosch. The researchers note: “We have found no evidence that automobile manufacturers write any of the code running on the ECU [Engine Control Unit]. All code we analyzed in this work was documented in documents copyrighted by Bosch and identified automakers as the intended customers.”

Discovering a hidden cheat

The researchers, led by University of California San Diego computer scientist Kirill Levchenko, faced a number of challenges in their quest to find the offending code.

Firmware images were gleaned from car-tuning forums and from an online portal maintained by Volkswagen for car repair shops. Documentation, in the form of so-called “function sheets,” was harder to come by. The function sheets were necessary to give the binary context, but the sheets are copyrighted by Bosch and generally not shared with the public. The research team ended up turning to the auto-performance tuning community again. These hard-core hobbyists and professionals share leaked function sheets so they can make aftermarket modifications to their cars.

“[T]he vehicle can switch to an operating regime favored by the manufacturer for real driving rather than the clean regime necessary to pass the emission test.”

Once the researchers were able to study the code running on the faulty diesels, they discovered that Volkswagen’s defeat devices were far more nuanced than anything found previously. Levchenko told Eurekalert that the “Volkswagen defeat device is arguably the most complex in automotive history.”

The researchers found that the cars assumed they were being tested in a lab until a sensor reading ruled out a lab test. At that point, “the vehicle can switch to an operating regime favored by the manufacturer for real driving rather than the clean regime necessary to pass the emission test,” the research paper noted (PDF).

Earlier, more basic versions of the software checked for three conditions to decide whether a lab test was being conducted. But by 2009, some firmware versions included 10 different checks for a lab test. The cars checked for things like how long and how far the car had been driven (in the US, emissions cycle tests last no more than 31 minutes and the car drives exactly 11.04 miles). Some cars also may have conducted steering wheel checks—if the steering wheel deviated more than 20° from neutral position, it was probably not being tested in the lab.

As each check ruled a lab test in or out, a different function of the Engine Control Unit—like fuel injection timing, how exhaust gas is recirculated through the vehicle, and how much urea should be injected into the exhaust—could respond.

Hidden in plain sight

In many of the function sheets, the code to cheat emissions tests was labeled as modifying the “acoustic condition” of the engine—a label that helped the cheat fly under the radar for as long as it did. But the “acoustic condition” wasn’t simply labeling legerdemain. Originally, “acoustic condition” code really did apply to the sound of the engine on startup.

The labelling checks out with a story in a legal complaint filed by the New York Attorney General in July 2016. The state’s top lawyer wrote that, as early as 2004, Audi was looking for a way to stop the rattling sound that diesel engines are known to make as the engine starts up. They allegedly solved the problem by injecting extra fuel as the engine initiated combustion. The extra fuel increased emissions. So, the “acoustic condition” was modified to discern when lab testing was occurring so that the Audis in question could pass their emissions tests and also have nice quiet start-ups that customers would find inoffensive.

According to that same 2016 complaint from the New York Attorney General, Volkswagen later struggled with building its selective catalytic converter—the company felt that the kind licensed by Mercedes-Benz was superior, and VW’s solution required an extra tank that stored “gallons” of Diesel Exhaust Fluid. Volkswagen then tried to build a “Lean Trap” for the catalytic converter to trap NO x , but engineers found that the car could only go 50,000 miles before the Lean Trap broke.

At this point, according to the NY AG, engineers turned back to that “acoustic condition” code for a little help to meet strict emissions regulations. Those engineers were also on a deadline to put forward a product that could compete with competitors’ diesel passenger vehicles.

That legal assessment of the defeat device scandal seems to have held up as the researchers analyzed the cars’ code. The VWs and Audis in question checked for a number of parameters at startup, and if a lab test was a possibility, the car would start with that assumption, enabling full emissions controls. The code permitted the car “to operate... as if two distinct personalities took turns controlling the vehicle,” the paper’s authors wrote.

The paper also notes that the researchers tested the diesel Fiat 500X because it used the same Engine Control Unit from Bosch as the Volkswagens and Audis did. There was no mention of the “acoustic condition” in the Fiat’s function sheet, but some undisclosed code was discovered controlling how the car regenerates its NO x Storage Catalyst (NSC).

“Unlike the Volkswagen defeat device, the FCA [Fiat Chrysler Automobiles] mechanism relies on time only, reducing the frequency of NSC regenerations 26 minutes 40 seconds after engine start,” the paper notes. In a normal system, the NSC reduces NO x emission by trapping it in a catalyst and then regenerating the catalyst as it gets full.

But regeneration hurts a car’s fuel economy numbers and puts a lot of load on the Diesel Particulate Filter (DPF). “By reducing the frequency of NSC regeneration, a manufacturer can improve fuel economy and increase DPF service life, at the cost of increased NO x emissions,” the researchers explained.

This problem? It’s an arms race.

To do a lot of their analysis, the authors of the paper developed a static analysis system that could scan auto firmware to look for defeat devices. They were largely successful with Volkswagens and Audis, but they stressed that more work has to be put into this problem. Staying ahead of automakers is difficult when they know precisely what regulators are looking for. Automakers, of course, stand to gain considerably if they can hide emissions cheats and deliver cars with performance superior to their competitors. Consumers will be excited to get better gas mileage—they won’t necessarily know that their car is spewing an outsized chunk of NO x into the air behind them.

The researchers also say that it’s high-time regulators dispense with the kind of lab tests that US and EU governments have required for years. Instead, some kind of active scan for illegal code needs to be developed. This problem, the paper notes, “drives a critical research agenda going forward that will only become more important as regulators are asked to oversee and evaluate increasingly complex vehicular systems (e.g., autonomous driving).”