The sharp spike in electricity prices in South Australia earlier this month certainly put the wind up some large energy users, and some long-time critics of renewable energy. But the really scary thing for the fossil fuel industry is that these price spikes don’t happen often enough any more.

You only have to go back a few years to find that such price spikes were so common they used to happen once every two days in the National Electricity Market, particularly in summer months. South Australia – because it was at the end of the grid and relied heavily on gas – was the worst afflicted. (See graph below on spikes above $5,000/MWh).

In fact, these price spikes used to be an important part of the business model for coal and gas generators. Spikes in demand, such as from air conditioners, would require more and more gas generation to be used to make sure the lights stayed on.

These spikes were so prevalent that about one-quarter of the annual revenue for the fossil fuel generators came from these demand and price spikes, because even the “cheap” coal generators used to get paid up to $14,000/MWh on these occasions. Their business models, and for some of them their ability to service debt repayments, banked on it.

(The networks also cashed in, using the surge in demand, and forecasts for a further increase in demand – that later proved to be completely wrong – as justification for a $45 billion spend on grid upgrades).

The proliferation of wind and solar has pretty much put an end to that particular lurk. Rooftop solar PV – there is now 5GW of it on home and business rooftops around the country – took out the demand and price peaks, making them smaller and pushing them later in the day.

Wind energy pulled down average prices too, including at night time when the industry had shifted most of the electric hot water service to underpin the market price for coal generators, with little else to power at that time of night. (Some of those hot water services are now being moved to the middle of the day).

The influx of renewable energy such as wind and solar is just the first stage of what is now universally described as the inevitable clean energy transition, replacing a business model based around large, mostly dirty, centralised generators to a new smarter, cleaner, more localised and faster responding grid.

The next stage will come with storage – be it built into the network, in homes and businesses, or attached to large-scale plants such as solar towers (with molten salt storage) or solar PV and wind farms with battery storage.

This will flatten out the process even further, removing much of the fuel price volatility and ushering in a new way of delivering electricity. And then more and more homes and businesses will use solar and storage to manage much, if not all, of their energy use. Half of all energy demand will be sourced locally, rather than from a central plant.

South Australia is at the forefront of this transition in Australia, and in some respects the world. Success here – and remember even the Australian Energy Market Operator says 100 per cent renewable energy is perfectly feasible – will have a cascading impact on the energy sector.

Little wonder, then, that the fossil fuel industry is pissed off. They can see the transition coming, and while some are making efforts to adapt, few can see how they can protect their traditional revenue base in a system that will likely be dominated by new players, new technologies and a new “energy democracy”.

Best, they reason, to slow it down. Protecting their existing revenue flows is the most obvious business strategy while they try to wrap their minds around what the future holds for them.

For that reason, the recent price spikes in South Australia and the rest of the country could not have happened at a more fortuitous time.

The fossil fuel industry and their supporters in the Coalition have taken advantage of a compliant, ignorant and/or lazy media to demonise wind and solar, and seek to stop the rollout of more wind farms in South Australia, while also discouraging other states from following the path of the South Australian government.

They have also been pushing for increased subsidies to protect the revenues of their gas plants, and fighting against rule changes that could accelerate the uptake of the competition, even if it does mean lower prices for consumers.

The real causes of the recent electricity price spikes, according to the Australian Energy Regulator, the South Australian government and independent analysts, have not been wind and solar at all, but the huge jumps in gas prices to record highs, cold weather in the eastern states, and supply constraints in the network.

The main interconnector between Victoria and South Australia has been constrained, gas prices in South Australia have been at record highs, double the price elsewhere and up to six times last year’s average, and as this occurred a large amounts of fossil fuel capacity was also not available.

Indeed, eight of the biggest coal plants in the country have had units offline recently, because of maintenance and repair at various times. In the industry, this is known as “outage season”. In the case of Loy Yang A and Loy Yang B, capacity was lost, sometimes at short notice, because of unexpected fuel supply and quality issues.

On top of this, the most efficient gas plant in South Australian was withdrawn from service just weeks before the Northern power station closed, and other gas plants were withdrawn at various times because either they were “not economic to run”, or because they “tripped” due to faults in their equipment.

The combined capacity of the unavailable fossil fuels was twice that of the total capacity of the wind farms. The AER, in the reports it has done to date, has been clear that the unexpected outages at coal and gas-fired power stations, along with the surging gas prices, have been responsible for the biggest spikes in power prices.

The South Australian government called it a “perfect storm”. Indeed, if someone was trying to manufacture a crisis (and we are not suggesting for a moment that anyone was) they couldn’t have organised it any better, failing having the actual lights go out (which they didn’t).

The critics, however, were relentless, and used the media to demonise South Australia’s “electricity experiment” as pure folly and a recipe for financial disaster.

The ideologues in the Murdoch press need little encouragement, but they got it anyway from the conservative Coalition, the fossil fuel lobby, and other vested interests that would prefer to protect the status quo of a heavily centralised grid controlled by just a few powerful players.

But far from being a sign of failure, the events of the past few weeks simply highlight the limits to fossil fuel generation, the danger of relying on variable cost fuels, and explains exactly why the South Australian government is pushing so hard to break its reliance on expensive gas and shake the market dominance of a few powerful energy incumbents.

South Australia energy minister Tom Koutsantonis says the national energy market is a train-wreck, not because of its growing reliance on renewables, but because it has refused to keep pace with new technologies, and takes no account of environmental outcomes.

He wants to break the market dominance, by reframing market rules, adding new interconnections, and introducing yet more renewable energy technologies.

He may get the chance to do so in the next few weeks, when he announces the results of the government’s low emissions tender. The hope is that the government will pick a solar tower and storage project in Port Augusta, but they may also choose a short-term bandaid and contract with the mothballed Pelican Point gas generator, if for no other reason than to bring more competition into the market in the short term. A second round of bidding may then offer opportunity for solar and storage, which is likely to take at least two years to be built.

Storage combined with renewable energy certainly signals the end game for fossil fuels. The gas lobby is seeking to lock in “capacity payments” to guarantee revenues into the future. But gas generators, even peaking plants, are slow compared to battery storage, and they are more expensive.

Storage also has the advantage of being able to deliver savings on network spending. Many want to introduce it to the grid now, but the rules are tilted against them. Proposed changes to the rules, such as the 5 minute settlement period, would encourage more battery storage and smart software and lower prices.

Even the regulators and market operators concede that the current structure results in “perverse” outcomes, but the incumbents are fighting the changes all the same.

Worse than that, the incumbents want the ambitious renewable energy targets of South Australia, and those of Victoria, Queensland and elsewhere to be cut short, and brought in line with the less ambitious national target.

And they appear to have strong support in the ruling federal Coalition, which has ridiculed state Labor targets of 50 per cent renewable energy or more as a “diabolical” idea. There is new talk that the generators will push for the current federal target to be further weakened.

But the market needs certainty, some sort of long-term vision that takes into account climate goals, as well as the reality that most of Australia’s fossil fuel fleet needs replacing, and that the obvious and cheapest substitute is solar and wind energy, and various forms of storage.