There was a familiar kneejerk reaction to last week’s announcement by the Australian Energy Market Operator (AEMO) that there are risks to our electricity supply after the scheduled closure of the Liddell coal-fired power station in New South Wales in 2022. The sight of the Prime Minister looking for options to keep Liddell open raises the spectre of further reflexive government intervention that can’t end well.

Governments, understandably, want to make sure the lights stay on. But now is the time for perspective, not panic. Because, as the latest Grattan Institute report – Next Generation: the long-term future of the National Electricity Market – shows, there are emerging challenges to the NEM that need dealing with. Make the right decisions now and a return to affordable and reliable electricity supply is on the cards.

Read more: The true cost of keeping the Liddell power plant open

The NEM is an energy-only market. This means that generators only get revenue when they sell their electricity into the market. All costs – including the capital costs of building the plant – need to be covered by the revenue they make when they sell electricity. Anyone who wants to build new generation capacity wants to be pretty certain that the market is going to deliver the revenue they need to cover their costs.

But right now no one is building any generation, unless it is government-backed renewables. This is despite a ripe environment for investment: high current and future prices in the wholesale market and the closure of old power stations. The result, as AEMO pointed out last week, is potential shortfalls in generation and potential blackouts in South Australia, Victoria and NSW over the next few years.

Much of the blame for this investment hiatus can be placed on politicians and the climate change policy mess that is creating so much uncertainty for potential investors.

Read more: Turnbull is pursuing 'energy certainty' but what does that actually mean?

But the rise of wind and solar power is also causing problems. Wind and solar energy have zero marginal cost: once the facility is built, the energy produced is essentially free. And they are intermittent suppliers: they don’t produce energy unless the wind is blowing or the sun is shining. So when wind and solar plants are operating, the wholesale price of electricity is forced down. This means there needs to be high prices – sometimes very high – when wind and solar are not operating. This price volatility makes investors nervous that they will not be able to cover the costs of building new generation.

Governments may be tempted to conclude that the market has failed. But intervention may be premature.

There are still five years until Liddell is scheduled to close. Just because a new coal-fired power station will not be built in time to fill the gap doesn’t mean the market cannot respond. Coal was never going to be the market response, given climate change risks. But new gas-fired generators, or batteries to store electricity, could be built in this time frame. Or the market could finally get its act together on what is called demand-response: that is, paying consumers to reduce their electricity consumption during periods of peak demand, so that less new generation is required.

Read more: Managing demand can save two power stations' worth of energy at peak times

There are no guarantees for government, however. The risks that the market won’t deliver the new generation that is needed are increasing. If nothing changes, Australia will need, in the words of AEMO, “a longer-term approach to retain existing investment and incentivise new investment in flexible dispatchable capability in the NEM”.

Many countries have responded to these same pressures by introducing a capacity mechanism. A capacity mechanism pays generators for being available, regardless of whether they actually sell electricity. Payments for capacity provide extra income for generators, giving them greater assurance that they will make enough revenue to cover their costs.

Any new market-based mechanism in Australia is likely to be better than the scattergun approach of various governments in recent years. Building Snowy 2.0, extending Liddell’s life, or providing state-based backing for new renewable generation might deliver the results needed. But the lack of coordination, planning and strategic thought that sits behind these policies means they probably won’t.

Getting it right

Our report suggests a better way. First, governments should give the market a chance. This means sorting out climate change policy, and quickly. Dithering about a Clean Energy Target, or arriving at a solution that cannot be supported across the political spectrum, will guarantee that investors’ hands remain firmly in their pockets.

Second, work should begin immediately on an additional capacity mechanism, so it is ready if needed. Capacity mechanisms are complex and take a long time to design and implement. There is no one-size-fits-all approach, so careful consideration needs to be given to how one would work in the Australian context.

Finally, AEMO should be asked to provide a more robust assessment of the future adequacy of generation supply. On the basis of this information, the newly formed Energy Security Board should make the judgement on whether an additional capacity mechanism is needed to make sure enough new generation is built.

It is understandable that politicians feel the need to act when faced with the threat of blackouts. After all, they are the ones who get the blame when the lights go out. But caution is needed. Capacity mechanisms are expensive; the peace of mind they bring comes at a price. A pragmatic and planned approach is the best way to ensure that, if a decision is made to redesign our electricity market, that decision is the right one.