ALMOST 150 years after photovoltaic cells and wind turbines were invented, they still generate only 7% of the world’s electricity. Yet something remarkable is happening. From being peripheral to the energy system just over a decade ago, they are now growing faster than any other energy source and their falling costs are making them competitive with fossil fuels. BP, an oil firm, expects renewables to account for half of the growth in global energy supply over the next 20 years. It is no longer far-fetched to think that the world is entering an era of clean, unlimited and cheap power. About time, too.

There is a $20trn hitch, though. To get from here to there requires huge amounts of investment over the next few decades, to replace old smog-belching power plants and to upgrade the pylons and wires that bring electricity to consumers. Normally investors like putting their money into electricity because it offers reliable returns. Yet green energy has a dirty secret. The more it is deployed, the more it lowers the price of power from any source. That makes it hard to manage the transition to a carbon-free future, during which many generating technologies, clean and dirty, need to remain profitable if the lights are to stay on. Unless the market is fixed, subsidies to the industry will only grow.

Policymakers are already seeing this inconvenient truth as a reason to put the brakes on renewable energy. In parts of Europe and China, investment in renewables is slowing as subsidies are cut back. However, the solution is not less wind and solar. It is to rethink how the world prices clean energy in order to make better use of it.

Shock to the system

At its heart, the problem is that government-supported renewable energy has been imposed on a market designed in a different era. For much of the 20th century, electricity was made and moved by vertically integrated, state-controlled monopolies. From the 1980s onwards, many of these were broken up, privatised and liberalised, so that market forces could determine where best to invest. Today only about 6% of electricity users get their power from monopolies. Yet everywhere the pressure to decarbonise power supply has brought the state creeping back into markets. This is disruptive for three reasons. The first is the subsidy system itself. The other two are inherent to the nature of wind and solar: their intermittency and their very low running costs. All three help explain why power prices are low and public subsidies are addictive.

First, the splurge of public subsidy, of about $800bn since 2008, has distorted the market. It came about for noble reasons—to counter climate change and prime the pump for new, costly technologies, including wind turbines and solar panels. But subsidies hit just as electricity consumption in the rich world was stagnating because of growing energy efficiency and the financial crisis. The result was a glut of power-generating capacity that has slashed the revenues utilities earn from wholesale power markets and hence deterred investment.

Second, green power is intermittent. The vagaries of wind and sun—especially in countries without favourable weather—mean that turbines and solar panels generate electricity only part of the time. To keep power flowing, the system relies on conventional power plants, such as coal, gas or nuclear, to kick in when renewables falter. But because they are idle for long periods, they find it harder to attract private investors. So, to keep the lights on, they require public funds.

Everyone is affected by a third factor: renewable energy has negligible or zero marginal running costs—because the wind and the sun are free. In a market that prefers energy produced at the lowest short-term cost, wind and solar take business from providers that are more expensive to run, such as coal plants, depressing power prices, and hence revenues for all.

Get smart

The higher the penetration of renewables, the worse these problems get—especially in saturated markets. In Europe, which was first to feel the effects, utilities have suffered a “lost decade” of falling returns, stranded assets and corporate disruption. Last year, Germany’s two biggest electricity providers, E.ON and RWE, both split in two. In renewable-rich parts of America power providers struggle to find investors for new plants. Places with an abundance of wind, such as China, are curtailing wind farms to keep coal plants in business.

The corollary is that the electricity system is being re-regulated as investment goes chiefly to areas that benefit from public support. Paradoxically, that means the more states support renewables, the more they pay for conventional power plants, too, using “capacity payments” to alleviate intermittency. In effect, politicians rather than markets are once again deciding how to avoid blackouts. They often make mistakes: Germany’s support for cheap, dirty lignite caused emissions to rise, notwithstanding huge subsidies for renewables. Without a new approach the renewables revolution will stall.

The good news is that new technology can help fix the problem (see article). Digitalisation, smart meters and batteries are enabling companies and households to smooth out their demand—by doing some energy-intensive work at night, for example. This helps to cope with intermittent supply. Small, modular power plants, which are easy to flex up or down, are becoming more popular, as are high-voltage grids that can move excess power around the network more efficiently.

The bigger task is to redesign power markets to reflect the new need for flexible supply and demand. They should adjust prices more frequently, to reflect the fluctuations of the weather. At times of extreme scarcity, a high fixed price could kick in to prevent blackouts. Markets should reward those willing to use less electricity to balance the grid, just as they reward those who generate more of it. Bills could be structured to be higher or lower depending how strongly a customer wanted guaranteed power all the time—a bit like an insurance policy. In short, policymakers should be clear they have a problem and that the cause is not renewable energy, but the out-of-date system of electricity pricing. Then they should fix it.