FROM his office window, Philipp Schröder points out over the Bavarian countryside and issues a Bond villain’s laugh: “In front of you, you can see the death of the conventional utility, all financed by Mr and Mrs Schmidt. It’s a beautiful sight.” The wind blowing across Wildpoldsried towards the Alps lazily turns the turbines on the hills above. The south-facing roofs of the houses, barns and cowsheds are blanketed with blue photovoltaic (PV) solar panels. The cows on the green fields produce manure that generates biogas which warms the Biergarten, the sports hall and many of the houses where the 2,600 villagers live, as well as backing up the wind and solar generators in winter. All told, the village produces five times more electricity than it needs, and the villagers are handsomely rewarded for their greenness; in 2016 they pocketed about €6m ($7m) from subsidies and selling their surplus electricity.

It hardly looks like the end of the world; but Mr Schröder, who works at Sonnen, an energy-storage firm, has a point. Many environmentalists want the world’s energy system to look like Wildpoldsried’s. And the things it is based on—subsidies for investment, very little spending on fuel, and moving electricity generation to the edge of, or off, the grid—are anathema to electricity markets and business models developed for the fossil-fuel age.

Few greens would mourn them. But the fall in utility revenues that comes with the spread of places like Wildpoldsried is not just bad news for fossil-fuel-era incumbents in the generation and transmission businesses. It is also becoming a problem for the renewables themselves, and thus for the efforts to decarbonise the electricity supply that justified their promotion in the first place.

In 2014 the International Energy Agency (IEA), a semi-official forecaster, predicted that decarbonising the global electricity grid will require almost $20trn in investment in the 20 years to 2035, at which point the process will still be far from finished. But an electricity industry that does not produce reliable revenues is not one that people will invest in.

Less dear, still disruptive

The fight against climate change has seen huge growth in the “new” renewables, wind and solar power, over the past decade, both in developed countries and developing ones. In 2015 governments poured $150bn into supporting such investment, with America, China and Germany taking the lead. But Wildpoldsried is still very much the exception, not the rule. In 2015 such sources accounted for only 7% of electricity generated worldwide. Over 80% of the world’s energy still comes from fossil fuels (see chart 1). In terms of reducing climate risks there is a long way to go.

The good news is that a decade of subsidy-driven growth has brought with it falling costs. Renewables are still on the pricey side in many places, but they are getting less so; in some places wind, in particular, is reasonably competitive. This suggests that their growth might soon need a lot less subsidy than it has attracted to date. Robust carbon prices would give renewables further advantages, but they have as yet proved hard to provide. The EU’s emissions-trading scheme is a perennial disappointment: still, hope springs eternal, as witness a recent attempt to persuade the new American administration of the benefits of a revenue-neutral economy-wide carbon tax devoted to providing $2,000 to every family of four in rebates.

But pushing renewables into the electricity market has had effects on more than their price; it has hit investment, too. In rich countries governments have imposed renewables on electricity systems that had no need for new capacity, because demand is in decline. Investment in supply beyond what the market required has produced gluts and pushed down prices. In America this has been somewhat masked by the shale-gas revolution, which has caused a bigger shift in the same direction. In Europe the glut of renewables is more starkly seen for what it is. Wholesale electricity prices have slumped from around €80 a megawatt-hour in 2008 to €30-50 nowadays.

The result has been havoc for the old-style utilities. Germany’s biggest electricity companies, E.ON and RWE, both split in two last year, separating their renewables and grid businesses from indebted and loss-making conventional generation. EY, a consultancy, calculates that utilities across Europe wrote off €120bn of assets because of low power prices between 2010 and 2015. Investment in non-renewables is very low. “Never in recent history has the deployment of capital been more difficult than it is right now within the energy industry,” says Matt Rennie, who analyses the global-utilities market at EY.

It is not just that efforts to shift to renewable power have added new sources of supply to an already well-served market. In an industry structured around marginal costs, renewables have a disruptive punch above their weight.

Electricity markets, especially those that were deregulated in the late 20th century, typically work on a “merit order”: at any given time they meet demand by taking electricity first from the cheapest supplier, then the next-cheapest, until they have all they need; the price paid to all concerned is set by the most expensive source in use at the time. Because wind and solar do not need to buy any fuel, their marginal costs are low. They thus push more expensive producers off the grid, lowering wholesale prices.

If renewables worked constantly that would not, at first blush, look like a problem for anyone except people generating expensive electricity. But renewables are intermittent, which means that in systems where the infrastructure was designed before intermittency became an issue—almost all of them, in practice—fossil-fuel, hydroelectric and nuclear plants are needed more or less as much as ever at times when the sun doesn’t shine and the winds don’t blow. And if such plants are shut out of the market by low-cost renewables, they will not be available when needed.

In the long run, and with massive further investments, electricity grids redesigned for systems with a lot of renewable energy could go a long way to solving this problem. Grids with lots of storage capacity built in; grids big enough to reach out to faraway renewables when the nearby ones are in the doldrums; grids smart enough to help customers adapt demand to supply: all have their champions and their role to play.

But long-run solutions do not solve short-term constraints. So for now countries with lots of renewables need to keep older fossil-fuel capacity available as a standby and to cover peaks in demand. This often means additional subsidies, known as capacity payments, for plants that would otherwise be uneconomic. Such measures keep the lights on. But they also mean that fossil-fuel production capacity clings on—often in particularly dirty forms, such as German power stations powered by brown coal, or backup diesel generators in Britain.

From dull to death spiral

Properly structured capacity payments make it sensible to invest in generators that can be switched on when renewable energy is not available. But what will make it sensible to continue investing in renewables themselves?

When they are a small part of the system, renewables are insulated from the effects that their low marginal costs have on prices, because as long as there are some plants burning fossil fuels the wholesale price of electricity will stay reasonably high. So utilities could buy electricity from renewable generators, often on fixed-price contracts, without too much worry.

But the more renewable generators there are, the more they drag down prices. At times when renewables can meet all the demand, making fossil-fuel prices irrelevant, wholesale electricity prices collapse—or sometimes turn negative, with generators paying the grid to take the stuff away (the power has to go somewhere). The more renewables there are in the system, the more often such collapses occur.

Rolando Fuentes of Kapsarc, an energy think-tank based in Saudi Arabia, claims the world is caught in a vicious circle: subsidies foster deployment of renewables; renewables depress power prices, increasing the need for financial support. Theoretically, if renewables were to make up 100% of the market, the wholesale price of electricity would fall to zero, deterring all new investment that was not completely subsidised. He calls this vicious circle the clean-energy paradox: “The more successful you are in increasing renewables’ penetration, the more expensive and less effective the policy becomes.”

Francis O’Sullivan, of the Massachusetts Institute of Technology, says the trend is already visible in parts of America with abundant solar energy. Utilities which are required to have renewables in their portfolios, such as those in California, used to offer companies investing in that capacity generous long-term contracts. But research by Bloomberg New Energy Finance (BNEF), a consultancy, shows that, as such utilities come closer to meeting their mandates, solar-power developers are being offered shorter-term fixed prices with a higher subsequent exposure to variable wholesale prices. That reduces the incentive to invest. Solar “cannibalises its own competitiveness away,” Mr O’Sullivan says. “It eats its own tail.”

At the turn of the century, according to the IEA, one third of investment in electricity markets flowed into “competitive” sectors that were exposed to wholesale prices; the rest went into regulated utilities, transmission grids and the sort of fixed-price contracts where the renewables got their start. By 2014 the share of investment in the competitive sectors was just 10% of the total. It is a fair bet that, the more renewables are exposed to competition by contracts pegged to wholesale prices, the more people will shy away from them as well.

Ever-lower capital costs, particularly in solar, could go some way to bucking this trend, making investments cheaper even as they become more risky.But if low-marginal-cost renewables continue to push prices down, there will come a time when private investment will dry up. As Malcolm Keay of the Oxford Institute for Energy Studies puts it, “The utility business model is broken, and markets are, too.”

Renewables do not just lower prices; when used by customers, they also eat into demand. Consider Australia. It has 1.5m households with solar cells on their roofs. There are a number of reasons for this. It is a sunny place; installing PVs was until recently generously subsidised; and electricity bills are high. In part that is to pay for some of the subsidies. In part it is because they pay for the grid, which has been becoming more expensive, not least because it has had to deal with a lot more renewables. The IEA says that in parts of southern Australia, grid upgrades have doubled network costs since 2008-09. Despite cuts to subsidies, Australian PV installations are expected to triple over the next decade.

When fewer people rely on the grid, there are also fewer left to share the costs. Phil Blythe of GreenSync, a Melbourne-based company that works with utilities to moderate the fluctuations of renewable energy, warns that his country faces an incipient “utility death spiral”. The more customers generate their own electricity, the more utilities have to raise prices to the customers that remain, which makes them more likely to leave the grid in turn. It won’t happen overnight, he says: but it is “death by a thousand cuts”.

From dromedary to duck

In California there is an icon for the effect that domestic renewables have on the demand for grid electricity, and thus on the revenues of utilities: it is called the duck (see chart 2). Every year more Californian consumers have solar cells. As a result, every year electricity demand during the day falls, and revenue falls accordingly. Similar effects are seen in Germany, where there are now 1.4m PV users—mostly domestic. It is one of the reasons—subsidies are another—why domestic electricity prices have stayed high there while wholesale prices have fallen. These home generators are not just reducing demand for grid electricity; often they are allowed to feed surplus power from their PVs into the grid, competing with other generators. In many American states utilities grumble about the “net metering” rate they are required to pay such people—especially in states like Nevada where they have been required to credit the electricity fed in at the retail price, rather than the wholesale price. And rooftop solar installations continue to grow, with 12 states more than doubling their deployment in 2016, according to BNEF. Businesses and industrial users are also becoming big consumers of renewable energy, which potentially reduces their dependence on the grid, and thus the amount they will pay for its services. The response to these problems is not to abandon renewables. The subsidies have helped costs of wind and solar to fall precipitously around the world. Competition is often fierce. Recent auctions for offshore wind farms in the North Sea and solar developments in Mexico and Abu Dhabi have shown developers slashing prices to win fixed contracts to supply clean electricity for decades to come. The “levellised cost of electricity” for renewables—the all-in cost of building and operating a plant over its lifetime—is increasingly competitive with fossil fuels in many places. Especially in sunny and windy developing countries with fast-growing demand, they offer a potentially lucrative, subsidy-free investment opportunity. But it does mean changing the way the world buys, sells, values and regulates electricity to take account of the new means by which it generates it. “Thinking of wind and solar as a solution by themselves is not enough. You need flexibility on the other side. It only makes sense if this is a package deal,” says Simon Müller of the IEA. Elements of that package are already appearing. Markets that sell commoditised kilowatt-hours need to be transformed into markets where consumers pay for guaranteed services. A lot more storage will be needed, with products like those of Sonnen in Wildpoldsried and the Powerwalls made by Tesla fighting for space in people’s homes. Smart grids bolstered by big data will do more to keep demand in line with supply.

It may not get all the way there In Wildpoldsried Mr Schröder dreams of electricity-users inviting friends round for a glass of wine to show off their new solar kits and batteries. “We’ll soon be at a point where people say, ‘You’re so yesterday. You get your power from the grid.’” But peer pressure is unlikely to be decisive. Bruce Huber of Alexa Capital, which helps fund renewable-energy investments, says business consumers are probably going to be more influential in driving the adoption of these technologies than households, because they will more quickly see how they might cut their bills by using demand-response and storage. “For the last 100 years everyone has made money upstream. Now the added value is coming downstream,” he says.

Waiting for enlightenment

Mr Huber likens the upheaval facing utilities to that seen in the telecoms industry a generation ago, when a business model based on charging per second for long-distance calls was replaced by one involving the sale of services such as always-on broadband. This is bad news for the vertically integrated giants that grew up in the age of centralised generating by the gigawatt. Jens Weinmann, of ESMT Berlin, a business school, names dozens of tech-like firms that are “nibbling” away at bits of utilities’ traditional business models through innovations in grid optimisation and smart-home management systems. With a colleague, Christoph Burger, he has written of the “big beyond” in which domestic energy autonomy, the use of the blockchain in energy contracts, and crowdsourcing of PV installations and other technological disruptions doom the traditional utility. Already, big Silicon Valley firms such as Google and Amazon are attempting to digitalise domestic energy, too, with home-hubs and thermostats.

But how this nibbling leads to a system that all can rely on—and who pays for the parts of it that are public, rather than private, goods—remains obscure. The process will definitely be sensitive to politics, because, although voters give little thought to electricity markets when they are working, they can get angry when prices rise to cover new investment—and they scream blue murder when the lights go out. That suggests progress may be slow and fitful. And it is possible that it could stall, leaving climate risks largely unabated.

Getting renewables to today’s relatively modest level of penetration was hard and very expensive work. To get to systems where renewables supply 80% or more of customers’ electricity needs will bring challenges that may be far greater, even though renewables are becoming comparatively cheap. It is quite possible that, as Mr Schröder predicts, Mr and Mrs Schmidt in Wildpoldsried will lay waste the world’s conventional electricity utilities while sharing Riesling and gossip with the neighbours. But that does not mean that they will be able to provide a clean, green alternative for everyone.