Last month's news that Volkswagen had been illegally rigging its diesel-powered cars to cheat on pollution tests has sparked all sorts of outrage. Hearings, lawsuits, fines, general opprobrium. And rightly so; the company's deception was appalling.

But there's a broader, more consequential problem here that a lot of coverage has danced around or hinted at only indirectly. So let's say it: Europe's promotion of diesel vehicles as a green transportation option has been a disaster thus far — for reasons that go well beyond the Volkswagen scandal.

Ever since the 1990s, European governments have been encouraging drivers to buy diesel cars as an alternative to traditional gasoline-powered vehicles. The rationale was simple: Diesel engines use fuel more efficiently, so the switch was supposed to reduce carbon dioxide emissions and help stave off global warming. Thanks to tax breaks and other incentives, diesel cars now make up one-third of Europe's fleet:

Europe's diesel push seemed like a perfectly sensible idea at the time. But they execution was badly botched, full of unintended consequences over the next 20 years.

One main drawback of diesel cars is that they can emit higher levels of other harmful air pollutants like particulates and nitrogen oxides. And those ended up being much harder to clean up than experts initially predicted. We now know that Europe's regulators have failed spectacularly to control diesel pollution, relying on weak rules and flimsy testing procedures. Lots and lots of automakers — not just Volkswagen — have been manufacturing diesel cars that emit far more gunk than they're supposed to. It's one reason why cities like London and Paris are still clogged with unhealthy levels of air pollution, causing thousands of premature deaths each year.

It also appears that Europe's diesel push didn't actually do much to help global warming, as one 2013 study by Michel Cames and Eckard Helmers found. The CO2 benefits from switching to diesel cars were overrated and likely offset by the extra soot the engines produced. On top of that, Europe's entrenched diesel industry has impeded progress on hybrid and electric car technologies that might have provided far deeper emissions cuts.

The whole episode is a sobering case study in how well-intentioned green industrial policy can go badly awry. So let's roll the tape and see what lessons we can learn from Europe's diesel problems.

Why Europe embraced diesel cars in the 1980s and '90s

When crude oil gets pumped out of the ground, it's sent to refineries to be turned into usable fuel. Those refineries typically distill the oil into lighter and heavier components. The lightest stuff includes gasoline. On the heavier side is diesel fuel, which contains more energy per gallon.

For most of the 20th century, automakers largely designed cars to run on gasoline, which was more flammable and combusted easily using sparks. Engines that could combust diesel fuel, using air compression, had been invented back in the 19th century, but they were noisier and belched more smoke, so they were mostly confined to large ships and trucks. Instead, diesel fuel was often used for heating and producing electricity.

But starting in the 1980s, French and German automakers began showing more interest in developing diesel cars. The reasons why have always been a little murky, although Cames and Helmers suggest it traces back to the OPEC oil crises of the 1970s. After global crude prices spiked, France decided to swear off using diesel for electricity and built a fleet of nuclear plants. Germany, similarly, switched from oil to natural gas for heating. When the crisis subsided, Europe's refiners were still producing lots of diesel with no buyers. So governments began urging automakers like Peugeot to look into diesel-powered vehicles.

By the late 1990s, diesel technology had improved dramatically, thanks to advances in fuel injection — common rail, particularly — that allowed the engines to run more quietly. The newer diesel engines were a technical marvel, operating more efficiently than their gasoline counterparts and using less fuel per mile traveled (and, importantly, emitting less carbon dioxide per mile). All they needed was a market.

Rising concern over global warming provided that push. In 1997, the European Union signed the Kyoto Protocol and committed to cutting its heat-trapping carbon-dioxide emissions 8 percent by 2012. The next year, the EU reached a landmark agreement with the continent's car manufacturers on reducing CO2.

At the time, there were lots of different paths Europe's automakers could have taken to green itself. They could've pursued direct injection technology for gasoline vehicles, making those engines more fuel-efficient. They could've ramped up development of hybrid-electric cars, as Toyota was doing in Japan. But European companies like Peugeot and Volkswagen and BMW had already been making big investments in diesel, and they wanted a climate policy that would help those bets to pay off.

Europe's policymakers obliged. The EU agreed to a voluntary CO2 target for vehicles that was largely in line with what diesel technology could meet. As researcher Sarah Keay-Bright later noted, these standards were crafted so as not to force Europe's automakers to develop hybrids, electric vehicles, or other advanced powertrains.

Meanwhile, European nations — including Britain, France, Germany, Italy, Spain, and Austria — had been cutting taxes on diesel car purchases and diesel fuel to promote sales, all in the name of thwarting climate change. Diesel sales soared. Back in 1990, just 10 percent of new car registrations in Europe had run on diesel. By 2011, that had climbed to nearly 60 percent.

Today, Europe's diesel cars are a public health problem

Diesel engines do have one notable pitfall. They may be more fuel-efficient and emit less CO2 than gasoline engines, but they also tend to emit higher levels of other nasty air pollutants, including soot, particulates, and nitrogen oxides (NOx). Heavy exposure to these pollutants can exacerbate heart and lung disease, trigger asthma attacks, and even cause premature death.

This was largely known back in the 1990s. Europe's policymakers simply considered the trade-off acceptable. "At the time, the prevailing belief was that climate change was the really hard problem and should be the priority, whereas we'd had experience improving air quality, so everyone assumed we could easily fix that issue later," explains Martin Williams, an air pollution researcher at King's College London who previously worked for Britain's environmental agency.

That assumption turned out to be wrong. When European regulators later moved to clamp down on NOx and other conventional air pollution from diesel vehicles, they failed badly.

Starting in 2000, the EU began ratcheting down the legal limits for NOx emissions (though Europe's diesel cars have always been allowed to emit NOx at a relatively higher rate than gasoline-powered cars). New vehicles were tested in laboratories, where cars were placed on giant treadmills, spun through a few exercises, and measured for pollution.

Trouble was, these tests turned out to be flimsy and easily gamed, explains John German of the International Council on Clean Transportation. Automakers could send cars to labs that were optimized for testing: stripped of excess weight, with the air conditioning turned off, and so on. Those test cars complied with the pollution limits just fine. But the cars that were actually being sold to consumers were quite different, with much higher emissions. (German notes that this sort of subtle gaming was technically legal, unlike Volkswagen's more elaborate deception, which involved illicit software that only turned pollution controls on during tests. Still, even if Volkswagen was the most flagrant cheater, it wasn't entirely alone.)

Few people realized the EU's pollution tests were badly flawed until 2010, when researchers started studying emissions from vehicles that were actually on the road. In one study, Williams set up roadside infrared detectors to measure NOx pollution from cars in seven British cities. What he found was shocking: Europe's newest diesel cars were emitting roughly as much NOx as older diesel cars from the 1990s.

In other words, 20 years of increasingly stringent air pollution regulations had done basically nothing to reduce diesel car emissions. "That was the most surprising part," Williams says.

Although overall pollution in Europe has gone down over time, diesel vehicle emissions remain stubbornly high. Today, Paris sometimes has smoggy days comparable to those in Beijing. London is struggling with unhealthy levels of nitrogen dioxide. Germany, Austria, and Ireland have NOx pollution well above the legal limits, with vehicles accounting for roughly 40 percent of that output.

The health toll is likely considerable. One recent study estimated that diesel pollution from cars, buses, and trucks in Britain caused 9,400 premature deaths in 2010 alone. It's difficult to pinpoint what fraction of those deaths might have been avoided if emission rules on cars had been strictly enforced all along, but that gives a sense of the stakes.

NO2 concentrations in Europe (areas above the healthy limit in red)

To be fair, European regulators have scrambled to improve the tests now that they've realized what was happening. But it remains to be seen whether Europe's diesel car industry will clean up its act.

Starting in 2014, the newest, most stringent emissions standards — known as Euro 6 — require diesel cars to emit no more than 0.08 grams of NOx per kilometer, an 84 percent reduction from 2000-era levels. By 2017, Europe's regulators will also start requiring on-road testing in addition to laboratory checkups. That's what the United States EPA already does, and it should eliminate the most obvious shenanigans (though, as the Volkswagen scandal showed, truly determined cheaters can be hard to catch).

In theory, it should be possible for diesel cars to meet these newest EU standards. The United States already has stricter emission standards around NOx and a few non-Volkswagen diesel models have met them successfully. One key control technology, known as selective catalytic reduction, involves injecting the car's exhaust with a mix of urea and water, which breaks the NOx down into harmless nitrogen, oxygen, and water molecules. Studies have found that a few diesel car models with this technology have low pollution even when tested on the road.*

Still, it's notable that Europe's automakers don't seem to think it will be so simple to comply with the EU's new standards, especially now that they can't game the tests. In October, the New York Times reported that Volkswagen, BMW, Daimler, Renault, Peugeot, and other manufacturers have been pleading with EU to relax the NOx limits by up to 30 percent.

Meanwhile, a 2015 study from the group Transport & Environment tested 10 of the newest European diesel car models on the road and found that 9 of them still exceed the Euro 6 standards for NOx pollution. The worst offender, an Audi, emitted NOx at 22 times the legal limit. The future of clean diesel is still an open question.

The irony: Europe's diesel push hasn't helped with global warming (so far)

If Europe's diesel surge over the last 20 years had helped mitigate climate change, then maybe (just maybe) you could argue that all this extra air pollution was worth it. Except here's a depressing plot twist: The climate benefits appear to have been negligible, at least so far.

In their 2013 paper, Cames and Helmers argued that Europe may well be worse off today, from a global warming perspective, than it would have been if automakers had just focused on improving gasoline-powered cars all along. And it's arguably much worse off than it would have been if automakers had started investing in hybrid-electric technology back in the 1990s.

The authors start with the chart below, showing that Europe's diesel cars may have once had a sizable CO2 advantage over traditional gasoline vehicles. But today that gap has narrowed considerably, as various technological advances have made modern gasoline engines nearly as efficient as diesel cars:

Carbon dioxide emissions from new cars in Europe and Japan

What's more, there are two subtle drawbacks to Europe's diesel cars that make them worse for climate change than they seem.

First, even if diesel cars emit less CO2 than gasoline vehicles, they emit a lot more black carbon, or soot, a pollutant that (we've recently learned) also contributes significantly to global warming. The precise accounting here is still subject to some dispute, but Cames and Helmers point out that black carbon emissions from diesel cars likely negate a big chunk of their CO2 advantage.

Second, remember that European countries encouraged diesel car adoption by slashing taxes on the diesel fuel itself, so that it was cheaper than gasoline. But as any Econ 101 student will tell you, cutting fuel prices gives people an incentive to drive more miles — and increase their overall emissions. This further chips away at any climate advantage diesel might have.

Add it up, and Cames and Helmers conclude that "the European diesel car boom did not cool down the atmosphere."

(By the way, this analysis doesn't even take into account the awful knock-on effects from Europe's subsequent policies to secure diesel fuel made from palm oil. Those biofuel mandates have led to widespread deforestation in Indonesia and Malaysia, further exacerbating global warming. The EU only recently scaled these policies back.)

One final coda here. Go back and look at the Japanese cars on the graph above. Back in the late 1990s, Japanese automakers were also thinking about cutting CO2, but they figured that diesel was a dead-end technology, since the cost of cleaning up the extra NOx pollution would be too high. Toyota, for one, started investing in hybrid vehicles, with the first Prius appearing in 1997. By contrast, Europe's automakers initially scoffed at electric powertrains, and none of them even bothered producing hybrids until Mercedes finally rolled one out in 2009.

Japan's decision now looks prescient, and Europe's looks shortsighted. Hybrids proved much greener: The newest Japanese cars now produce roughly 10 percent less CO2 per mile than Europe's diesel vehicles do, on average. And Japan's automakers may well be better set up for the future. Many experts now believe that transportation will eventually have to become electrified if we want truly deep reductions in emissions. But European automakers, stuck on diesel for so long, are scrambling to catch up.

Europe is now shifting away from diesel — but it's not easy

Because of the Volkswagen scandal, and the staggering levels of pollution in cities like Paris and London, Europe's policymakers are now beginning to rethink their fondness for diesel cars. But it's a hard technology to give up. Path dependency is a hell of a drug.

In diesel-loving France, Ségolène Royal, the environment minister, recently said that the country would consider phasing out preferential tax breaks for diesel over the next five years. "We need to start preparing our move out of diesel," she reportedly told France 5 television. But Royal has also bristled at any suggestion that France should act too quickly. After all, diesel cars still account for more than 60 percent of all European sales for Renault and Peugeot, two major local manufacturers. They can't just unwind those positions overnight.

Volkswagen, for its part, is also engaged in a bit of soul searching around diesel (at least, when it's not recalling its 11 million law-breaking cars and dealing with criminal probes). The company recently announced that it would take a major plunge into electric powertrains, creating a standardized architecture for a new wave of plug-in cars. "The Volkswagen brand is repositioning itself for the future," said executive Herbert Diess. Yet this shift won't happen overnight. Because VW spent so many years dreaming up ways to evade regulators and sell diesel cars, it's now playing catch-up to electric car companies like Tesla.

In the meantime, Europe's smoggy cities still have to grapple with the fallout from all those older, polluting diesel cars still on the road — cars that will stick around for years.

If automakers can start manufacturing more truly clean diesel cars, that would help, but it will still take time to replace the older cars on the road. So, in addition, cities like London, Stockholm, and Milan are now experimenting with "emission-free zones." In the future, drivers with heavily polluting diesel clunkers could have to pay extra to travel downtown in certain cities. It's likely to be a slow, messy process.

Energy bets can sometimes go horribly wrong. How do we minimize that risk?

Europe's misadventure with diesel cars is a great case study in how energy policy can go very badly awry, and no doubt there's a long list of lessons to draw. I'll just mention a few big ones:

First, Europe's complete failure to regulate NOx and other diesel pollution for 20 years is worth studying closely. Not only had regulators designed shoddy emissions tests — partly at the behest of industry lobbying — but it took two decades to even realize the tests were failing. Crafting ambitious environmental rules is all well and good, but without ample enforcement and monitoring, those rules are basically pointless. (Note that China is now pursuing ambitious climate policies of its own, but experts keep muttering that enforcement will be a big challenge — a caveat that deserves serious attention.)

Another possible takeaway is that, if we really want to reduce greenhouse gas emissions from the transportation sector, liquid fuels are looking like a dead end and electrification might be the best path forward. There's no shortage of experts who believe that, and certainly that's the lesson Volkswagen seems to be drawing from this fiasco. That said, it's worth noting that Toyota, which was ahead of the pack in developing hybrid electrics in the late 1990s, actually doesn't believe battery-electric cars are the future and is pouring R&D into hydrogen cars instead.

Which brings us to the third takeaway. The future is hard to predict. Diesel cars seemed like a reasonable idea in the 1990s and a disaster today. That suggests that policymakers should have a lot more humility when crafting energy policy. Maybe battery-electric cars will win out, or maybe it'll be hydrogen, or maybe it'll be something else entirely. (Heck, perhaps diesel cars that are genuinely clean could play a role in reducing CO2 emissions.) No one knows for sure.

So one approach here might be to pursue technology-neutral policies focused on preferred outcomes — say, tightly enforced standards that require lower emissions — rather than favoring specific industries and technologies just because they happen to seem promising at that moment in time.

This conundrum is likely to come up again and again. For years, governments have been laying down big bets on emerging clean energy technologies. France did it with nuclear power in the 1970s and '80s. Germany did it with wind and solar power in the 2000s, through feed-in tariffs. The United States has done it with corn ethanol in the past decade.

Done right, this sort of government support can be valuable, helping useful new energy options break into the mainstream against entrenched competition. But there's also a huge risk that governments will end up gambling on badly flawed technologies that then become the entrenched competition — and prove impossible to get rid of. The US arguably made that mistake with ethanol, which has had unintended ripple effects on the food supply and deforestation that are proving politically difficult to untangle. The drive for diesel looks like it belongs in that category, too. It's not a story we'd like to keep repeating.

* Update: Just to clarify, there are a few clean diesel car models, such as BMW's 328d sedan, that appear genuinely capable of meeting the latest, most stringent pollution rules and tests in both Europe and the United States. (My original wording in this section was unclear.)

Further reading:

-- Michel Cames and Eckard Helmers' 2013 paper "Critical evaluation of the European diesel car boom — global comparison, environmental effects and various national strategies" is well worth reading for the details on the diesel fiasco.

-- The Guardian's Jon Vidal also wrote a great piece exploring the health effects of the rise of diesel in Europe.

-- The International Council on Clean Transportation has done some of the best work documenting the divergence between tests and real-world emissions for cars in Europe. (This work helped them uncover Volkswagen's deception.) This report offers a great summary. Note also that Europe's cars, both gasoline and diesel, tend to get worse fuel economy than tests suggest.