John German had not been looking to make a splash when he commissioned an examination of pollution from diesel cars back in 2013. The exam compared what came out of their exhaust pipes, during the lab tests that were required by law, with emissions on the road under real driving conditions. German and his colleagues at the International Council on Clean Transportation (ICCT) in the US just wanted to tie up the last loose ends in a big report, and thought the research would give them something positive to say about diesel. They might even be able to offer tips to Europe from the US’s experience in getting the dirty fuel to run a little cleaner.

But that was not how it turned out. They chose a Volkswagen Jetta as their first test subject, and a VW Passat next. Regulators in California agreed to do the routine certification test for them, and the council hired researchers from West Virginia University to then drive the same cars through cities, along highways and into the mountains, using equipment that tests emissions straight from the cars’ exhausts.

Dirty lies: how the car industry hid the truth about diesel emissions – podcast Read more

It was clear right away that something was off. At first, German wondered if the cars might be malfunctioning, and he asked if a dashboard light had come on. That didn’t really make sense, though – the cars had just passed the California regulators’ test. His partners thought there might be a problem with their equipment, and they recalibrated it again and again. But the results didn’t change. Nitrogen oxide (NOx) pollution from the Jetta’s tailpipe was 15 times the allowed limit, shooting up to 35 times under some conditions; the Passat varied between five and 20 times the limit. German had been around the auto industry all his life, so he had a pretty good idea what was going on. This had to be a “defeat device” – a deliberate effort to evade the rules.

“It was just so outrageous. If they were like three to five times the standards, you could say: ‘Oh, maybe they’re having much higher NOx emissions because of the high loads,’” or some other external factor. “But when it’s 15 to 30 times the standards, there is no other explanation,” he says. “It’s a malfunction or it’s a defeat device. There’s nothing else that could possibly get anywhere close.”

German wasn’t ready to level such a serious accusation against a huge company such as Volkswagen, so he kept quiet while the research moved forward. Much later, his boss was surprised to learn how early he had suspected the truth. “He said: ‘You knew there was a defeat device? Why didn’t you tell me?’” The answer was simple. “We’re an $8m organisation. VW could have squashed us like a bug.”

German and his colleagues pressed ahead with their work and, when the study was finished, they posted it online. That was May 2014. He was still nervous, so the council didn’t issue a press release, nor did the report name the manufacturer. As a courtesy, he sent a copy to someone he knew at Volkswagen, noting “by the way, Vehicles A and B are yours”. German’s group also forwarded the findings to the US Environmental Protection Agency (EPA) and California’s Air Resources Board (Carb). “We were definitely scared. We wanted EPA and Carb to take over.” After the results were posted, he would email the agencies now and then. No one replied, and having spent more than 13 years at the EPA himself, he knew what that meant.

The regulators were investigating. And while they struggled to determine what was causing the discrepancy between pollution in the lab and on the road, Volkswagen executives quietly debated their next move. After months of foot-dragging, Volkswagen promised to remedy the problem, which it blamed on a technical glitch. It began recalling cars, updating the software in hundreds of thousands of them.

Months later, California ran new tests. Emissions were still far over the limit. Now regulators wanted to see the software controlling the vehicles’ pollution systems. And they made an extraordinary threat to get it: if Volkswagen did not turn over the code, it would not get the approvals it needed to sell cars in California and a dozen states that used its standards. The EPA threatened to withhold certification for the entire US market. “That,” German says, “was when VW came clean.”

Dieselgate, as it became known, exploded into one of the biggest corporate scandals in history. Over almost a decade, Volkswagen acknowledged, it had embedded defeat devices in 11m cars, mostly in Europe, but about 600,000 in the US. The software detected when emissions tests were being run, and pollution controls – components inside the engine that reduce emissions, sometimes at the expense of performance or fuel consumption – worked fine under those circumstances. But outside the lab, the controls were switched off or turned way down, and NOx levels shot up as high as 40 times the legal limit. With mind-boggling gall, Volkswagen had even used the software update it was forced to carry out to improve cars’ ability to detect when they were being tested.

And, as it turned out, Volkswagen wasn’t the only one evading the law. Less flagrantly, but to similar effect, the vast majority of diesel cars were making a mockery of emissions rules. In the wake of the revelations in the US, European governments road-tested other big brands too. In Germany, testers found all but three of 53 models exceeded NOx limits, the worst by a factor of 18. In London, the testing firm Emissions Analytics found 97% of more than 250 diesel models were in violation; a quarter produced NOx at six times the limit. “As the data kept coming in, our jaws just kept dropping. Because it is just so systematic, and so widespread,” German says. “VW isn’t even in the worst half of the manufacturers.” With a few honourable exceptions, “everybody’s doing it”.

In the US, where only around 2% of cars are diesel, the rule-breaking had an impact. But the health consequences have been far more severe in Europe, where drivers had been encouraged for years to buy diesel cars – when the scandal broke, they accounted for more than half of all sales. In 2015 alone, one study found that failure to comply with the rules caused 6,800 early deaths. To put it more plainly, tens of thousands of people had died because carmakers felt so free, for so long, to flout the law.

Of course, the painful light cast by the scandal didn’t just expose corporate wrongdoing. It also made visible a failure that is just as distressing. Across Europe, including in Britain, governments responsible for enforcing the law and protecting their people’s health had utterly neglected to do so. The fact of the matter, German explained to me, is that European air quality regulators don’t have the muscle or the resources their US counterparts have long possessed. European countries have never built the enforcement capability needed to give teeth to pollution rules. Governments, he says, “don’t seem to be able to do anything about it, in most cases don’t even seem to want to do anything”.

Facebook Twitter Pinterest Air pollution hanging over London. Photograph: Stefan Rousseau/PA

While the US is, in so many ways, an environmental laggard compared to Europe, air quality is a glaring exception. The EPA has, over the years, built up tremendous legal and technical expertise. At least until its evisceration in the Trump years, the EPA was known for its diligence in supplementing regulations with circulars and advisories that precisely defined every term, clarifying ambiguities and laying out what was allowed and what was not. The result was a system that, if not watertight, was a lot less leaky than elsewhere. In Europe, while the rules might look similar, no one goes to the trouble of making clear exactly what they mean, so polluters provide their own interpretations. Its atrocious air offers a cautionary tale that those undermining US regulation would do well to heed.

I have lived in London for 18 years, breathing the diesel fumes that foul the city’s air. I can smell the emissions when I’m out running errands. After a few minutes on a busy road, I often have a mild headache. German has given me a full understanding of the pollution that has obsessed and infuriated and terrified me all this time. I see now that blame does not lie just with the misguided, turn-of-the-century decision to nudge drivers toward diesel. Nor only with car companies’ rule-breaking. The failure of so many governments to enforce the law is the missing third piece of this puzzle. What I understand now is that the people we entrusted with the power to protect us essentially decided not to bother. Instead, they have allowed carmakers to spew whatever they want into our air.

That, at bottom, is the explanation for the filth that leaves grit on my teeth and a sour taste in my mouth, and puts me and my family and my neighbours at risk of heart trouble, dementia and early death in many forms. Together, the mistake and the cheating and the negligence are why the streets we walk every day in London are fouled by noxious fumes.

How could this have happened in countries that are among the wealthiest in the world, on a continent whose name is a byword, elsewhere, for environmental progressivism? Not long after meeting John German, I was in Berlin at the European office of the ICCT – the US arm of which German works for – to see his colleague Peter Mock.

As Mock spoke, I began to absorb the particulars of Europe’s stunning failure. It starts with an enforcement structure that almost seems designed to let violators through. The European commission sets the rules on how much pollution a car is allowed to produce. But the job of enforcing them falls not to Brussels, but to national governments. And a car company preparing to release a new model can choose which country certifies it; every EU nation must then honour the approval. A savvy carmaker opts for a place where it provides lots of jobs, where officials are likely to be pliant.

The national enforcement agencies, for their part, are generally understaffed, poorly funded and lacking in technical expertise. Britain is an exception, but in most nations these weak agencies don’t even test cars themselves. About a dozen individual vehicles must be checked before a new model is approved, and the tests are often run by outside contractors. When they are done, the manufacturers hand the paperwork to regulators, and the results, says Mock, are usually accepted with little question.

What’s more, the specifics of the tests – speed, acceleration and so on – are publicly available. So a manufacturer can build its cars to produce little pollution under those particular conditions and a lot more the rest of the time. This is the key to a question that has been nagging at me. I know that many diesel brands, not just Volkswagens, shatter emissions limits. Yet most companies don’t have VW’s legal troubles.

There is another route those companies take: programming pollution controls to turn off when the weather is too hot or too cold, when a car is just being started or is speeding up or slowing down or climbing a hill – conditions they frame as extraordinary, but account for a big chunk of driving time. If challenged, the companies can cite a legal loophole, claiming the switching off is necessary to protect engines.

Facebook Twitter Pinterest An air pollution warning in London last year. Photograph: Guy Bell/Alamy

Now, at last, European regulators have begun requiring cars to be tested on the road, not just in the lab. But the real problem, to my mind, is even bigger: it seems clear that the flaws in European nations’ enforcement are more fundamental than the particulars of one testing method. The problem is the system itself, which is riddled with weakness and ripe for abuse. Politicians have begun, post-Dieselgate, to tighten it, but it remains a system designed under the gaze – and the lobbying pressure – of a powerful industry.

I learned to my astonishment that some in power knew about the consequences all along. I spoke by phone to Martin Schmied, an official at Germany’s federal environment agency. His department, he told me, had been taking cars on the road for 25 years to measure emissions – and publishing the results. Year after year, they found diesels producing NOx above the legal maximum; six times, in one recent test. I asked him to clarify: Germany’s government, and anyone who read its public reports, has known for decades that automakers were flouting the rules? Schmied responded that as long as emissions went down when limits were tightened, his department didn’t mind they were many times higher than allowed. “We publish this data,” he said. “In principle, this is nothing new.”

So Germany knew. Perhaps other governments did, too. Many of its people, though, did not. I certainly didn’t. Nor did the buyers of millions of diesel cars. Nor the hundreds of millions of people who breathe the air they taint, trusting for so long that companies were following the law – and that governments would catch them if they didn’t. David King, Tony Blair’s scientific adviser, told me he gave his support to the tax changes that would put so many diesel cars on to British roads because he believed they would meet emissions limits.

The diesel cheating scandal is in some sense a failure of innovation – yet another symptom of carmakers’ desire to stick with what they know, with the cars that reliably deliver profits. That caution is surely at the root of why European manufacturers pushed governments looking to shrink carbon footprints to turn to diesel, rather than, for example, hybrids such as those that Honda and Toyota had already put on roads by the late 1990s. With their vast resources and the marketing muscle to bring consumers along, who knows what Volkswagen and the others could have come up with. We have all paid the price for their decision not to try.

Today, glimmers of a different future are in sight, as electric cars begin moving from niche to mainstream. There are challenges, to be sure: the need for better batteries, more charging points and enough power to keep cars supplied. But those are obstacles that can be overcome and the technology is advancing quickly.

The stakes are higher than ever. Today’s cars – petrol, if not diesel – are, by some measures, 99% cleaner than the barely regulated ones of 1970. But while governments have taken aim at the pollutants that harm our bodies, they have hardly begun to target the one that is warming our planet: until recently, no auto regulation sought to reduce carbon dioxide. So it has climbed along with the number of cars on the road, the miles they drive and the gallons of fuel they burn.

Today, electric vehicles look like the best way to slash both sorts of pollution, another place where the goals of a healthy climate and healthy bodies converge. Electricity by itself is no guarantee of climate friendliness. But it is a necessary prerequisite to powering cars from clean sources such as wind and solar.

Electric cars are not a cure-all. While they don’t create exhaust emissions, their brakes and tyres give off tiny, toxic particles as they wear. The energy needed to manufacture them, and the raw materials used in their bodies and batteries, will be unsustainable if car ownership keeps increasing.

For now, that relentless rise frames everything. The number of vehicles in the US has more than tripled since 1960; in the UK, there is one car for every two people. And the biggest growth is now in developing nations such as India and China. If they follow the path we have taken, the world could go from about 1bn cars today to more than 3bn by 2050. What is really needed is not just a slowing of that growth, but fewer cars altogether, of any sort. It is a goal that is reachable if we reorganise the places we live to be denser, more pedestrian- and bike-friendly, with public transportation – and newer options such as car-sharing – that are convenient and affordable.

Still, cleaning up the vehicles we do drive is crucial. As in the past, the best hope comes from companies willing to put in the money and brainpower needed to do it. And, as ever, a lot of powerful players are deeply invested in the old ways of doing things, so progress has sometimes been grudging.

But this time, there are some important new forces at work. One of the biggest is China. The country’s leaders have recognised the urgency of confronting their pollution problem, and they are eager to dominate the industries needed to do so, electric cars very much included.

China’s hunger for clean cars – along with its willingness to put big money into its top priorities – is transforming the sector in ways likely to affect us all. It is by far the world’s largest market for cars, and its demand for electric ones is ramping up fast, so it is now the biggest buyer of those, too. The government is aiming to get millions more on to roads, offering rebates to drivers who buy electric and telling big multinational carmakers that if they want to do business in China, they have to hit ambitious targets for low- and zero-emission models. That aggressive push is sure to accelerate trends that are already underway globally: falling prices and advancing technology, particularly better batteries that increase range and charging speed.

In the US, California is out in front, pushing to get millions of zero-emissions cars on to roads. Norway is another leader, and Europe more broadly is also taking to electric cars – because of drivers’ anger over the diesel scandal, a wider understanding of air pollution’s threats, and the need to confront climate change.

It feels fitting that, to get a glimpse of a cleaner future, I must barrel down a highway crowded, at the tail end of a northern California rush hour, with the gas-guzzling SUVs and big diesel trucks of the present. I was headed to Tesla’s factory in Fremont, on the edge of Silicon Valley. Indeed, it feels inside as much like a tech operation as a traditional manufacturing one, not post-industrial, but post-fossil fuel, post-dirty. The vast building, a mile-and-a-quarter long, is surrounded by acres of parking. I’ve been warned to allow at least 15 minutes to find a spot – which proves a wise suggestion – and except for a few plug-ins at the front, nearly all the cars here are the traditional kind.

Inside, a clutch of Tesla enthusiasts had gathered for this morning’s tour. As our guide, Kim, in black slacks and T-shirt, her grey hair long and loose, led us toward the factory floor, she began a well-practised patter, mixing facts and figures with jokes and shout-outs to fellow Tesla owners. Her enthusiasm seemed genuine. “Each and every day you drive your Tesla, each and every mile, you are helping to save the world,” she told us.

In a small demonstration area, Kim passed around some of the materials used to make these vehicles. First came a small cylindrical battery, the size of two or three AAs; about 7,000 of them, she explained, are packed together into the pale green, flat-bottomed case that formed the undercarriage of a nearby display car. Next came a metal ingot, then a tub of black plastic beads that reminded me of the more colourful ones my artsy daughter arranges into heart and star shapes for me to melt together with an iron. When we climbed into a long trolley, Kim put on a headset and got behind the wheel.

As Kim drove and talked, I gazed at a sea of metal parts, stacked on shelves in rows hundreds of feet long. Some were recognisable as wings or doors; others had shapes whose meaning I couldn’t discern.

Facebook Twitter Pinterest A Tesla Model S being assembled by robots in Fremont, California. Photograph: The Washington Post/Getty Images

A press as big as a small building turned giant rolls of aluminium into body panels; Kim said its foundation extends three stories down. Several workers were inspecting pieces as they emerged on a conveyor belt. At many stations, no one was present – just red robotic arms, quietly sliding back and forth, up and down, spinning and turning and lifting, riveting and welding. But while the employees were scattered, there were quite a lot of them, all told. Many sat or stood at computers in office-like clusters of desks and tables that opened on to the larger work floor. We passed a cafeteria with a salad bar and coffee counter; it was open to the factory floor, and it was filled with workers chatting and eating.

As we drove on, I began to see the shells of cars, scores of them lined up in rows. In the “hang-on area”, doors were being installed and seats were stacked nearby. Tyres, too, then more car bodies, freshly painted ones. As Kim bade us farewell (“Hopefully I showed a lot of the magic and the mystery around your car,” she said), I finally realised what was missing – the part I hadn’t seen, one so familiar it has taken me until now to clock its absence. There are no exhaust pipes on these cars.

Tesla – with its sleek style and big ambitions, its well-publicised troubles, and a CEO, Elon Musk, who one columnist called “the id of tech” – has taken on outsized symbolism as the representative of an industry hoping to jump from its infancy straight into adolescence and beyond. Its cars drive smoothly, require little maintenance and are replete with clever touches such as door handles that pop out when a driver approaches and large touchscreens in place of old-fashioned dashboard controls. Teslas’ desirability and the hype around its cars spring from buyers’ belief that they offer not just a replacement for traditional cars, but something far superior. The company has struggled to manufacture fast enough to fulfill promises to customers, and financial problems at times have made its future look cloudy. But it has clearly succeeded in providing a proof of concept, settling once and for all the question of whether electric cars can be both reliable and elegant, or able to give drivers what they need and what they want. In doing so, it has prodded others to follow. And in the big picture, that matters far more than the fortunes of this one company.

Musk is a brash South African immigrant who, in many ways, epitomises what the US has always imagined itself to be: daring and hard-driving, ready to dream big and take risks. He is sometimes compared with Apple’s Steve Jobs, but, with typical tech-world self-belief, Musk sees his mission as loftier than making beautiful gadgets – he wants to put technology to work solving humanity’s most pressing problems. He has made more progress than many expected. SolarCity is his bid to expand clean power use, Powerwall his push for the batteries that make renewables reliable. With Tesla, he wants to remake transportation, and he bet – correctly – that he could beat a vast but moribund industry to it.

The plastic backlash: what's behind our sudden rage – and will it make a difference? Read more

Walking out of the factory, I saw an oversized US flag billowing in the distance. Right beside it, another banner bore the oil giant Chevron’s familiar red-and-blue logo. You don’t have to look far from Tesla’s bubble to see the fossil-fuel economy is still going strong. And in case I needed another reminder that electric cars are still a tiny speck in a huge gas- and diesel-powered sea, a truck carrying half a dozen shiny Teslas pulled up behind me as I headed toward the highway. Despite its cargo, the truck was the traditional sort, dirty and lumbering, almost certainly diesel.

So there is a long way to go before technology fulfills its promise on a scale big enough to matter. But Tesla, and others taking up the gauntlet it has thrown down, offer a peek at what is possible. Less important than whether that future is delivered by Silicon Valley or Detroit, Beijing or Wolfsburg is that it dawns quickly. Innovation today offers the hope of a revolution that finally takes us where we need to go.

This an edited extract from Choked: The Age of Air Pollution and the Fight for a Cleaner Future by Beth Gardiner, published by Granta on 4 April and available at guardianbookshop.com

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.