It was the head of the biggest electric network operator in the world, China State Grid, that summed up best the challenge of moving to a high renewable energy grid: It is not so much a technical problem, but a cultural one.

In other words, there are those who say it can be done, arguing that it offers a smart, cleaner and ultimately cheaper and more reliable alternative. And there are those who say it can’t be done, and are reluctant to adopt the new technologies and the new ways of managing a complex electricity grid.

In Australia in the past few weeks, we have been getting a clear signal as to which authorities fall into which camp, and the obstacles facing those who want to get on with the job and go with the technology, rather than fight it.

There is, inevitably, the politics, led by the federal Coalition, railing against the “reckless pursuit” of wind and solar and yet, at the same time, drumming up huge ideas for massive pumped hydro schemes, a sure sign that they see more wind and solar as inevitable.

And there is institutional resistance. The Australian Energy Market Commission, which sets market rules, last week released a document which painted a view of Australia’s energy market nearly as dystopian as Donald Trump’s inauguration speech, the one that prompted former president George W Bush to note at the time: “That was some weird shit.”

And so was the AEMC’s. Its full document is a thorough appraisal of the events of 2015/16, but the media release was another thing altogether: painting a dark picture of energy shortages, risky additions of wind and solar, lost inertia, reduced reliability and the threat of blackouts – comments that were readily picked up by the green-baiting Murdoch media.

Ivor Frischknecht, the CEO of the Australian Renewable Energy Agency, has said on several occasions in the last few weeks that it is clear that the technologies exist for transition to a renewables-based electricity grid. It is only old rules and regulations that are getting in the way and preventing it from happening.

It’s a view that is now widely shared. The CSIRO and Energy Networks Australia, in their ground-breaking Network transformation roadmap, speak of the critical important for rules and regulations to catch up with technology, lest the changes and cost reductions in solar, storage, and software becomes so rapid that the industry is unable to catch up.

Their two-years of research found a zero emissions grid could be put in place, based largely around renewables and with a special focus on consumer-owned solar and storage, and save consumers more than a $100 billion by 2050.

That would be at least some recompense to those consumers, who are clearly the biggest losers from the creation of the National Electricity Market two decades ago, and its failure to check the spending of the networks or the pricing power of the gentailers.

The consumers are now paying ridiculous prices from electricity still mostly delivered by now mythical “cheap coal”, and are facing even more rises in coming months.

Yet, as Accenture points out in a separate report, these consumers now have the technologies to be masters of their energy destiny, driven by concerns about sustainability, energy independence and simple economics.

When the cost of solar and storage is likely to be half the cost of grid power, as some networks recognise it will be, the economic modelling behind this grid concoction has a major, major problem, one that rivals the disruption posed by the internet and digital technology.

And because this is a heavily regulated and essential service, the challenge is not just to the incumbents but the regulators and rule makers.

Accenture warns that unless the industry changes quickly, there will be hell to pay in their boardrooms, and consequences everywhere. To do that, they need the rules to be changed, and to be changed quickly.

The Grattan Institute added to those calls on Monday, saying that urgent market reforms and rule changes are needed to ensure reliability of supply. It is hard to find anyone in the industry who disagrees with this statement.

The irony is that it is the AEMC that is charged with making and adjusting these rules, which makes its position on the risks to energy security all the more galling for many, given it has done so little to make the grid fit for purpose, either rejecting new proposals, or kicking them endlessly down the road.

The Australian Energy Market Operator has grown so frustrated with the situation that in its submission to the Finkel Review it asked to be allowed to take responsibility of many of the rule changes itself, so it can rapidly adapt the markets to the changing technologies and dynamics.

This call is likely to be intensified under its new CEO, the reforming Audrey Zibelman, and it was notable that last week AEMO and ARENA teamed up to drive a pilot on the use of demand response, an obvious and relative cheap solution to dealing with peak demand, and a lot cheaper and cleaner than building new peaking generators.

Zibelman knows it will work, because she has seen it operating effectively in markets throughout the world, including the one in the US where she used to manage New York’s radical shift in energy policy.

“There is often skepticism about change,” she told RenewEconomy last week. “This (trial) is a good way to show this technology can work. And when we have done that we can get it into the market and modify the market rules. Technology is changing. We have to look at the market design, to ensure it attracting the right sort of investment.”

It just so happens that demand response has been one of many initiatives presented to the AEMC (way back in 2012) that were rejected or delayed, with the rule maker arguing that there was sufficient demand response in the system. Clearly not, given the enforced load shedding that occurred across the country last summer.

But demand response is just another example of the number of initiatives that the incumbent fossil fuel industry has managed to have killed or shrunk: think carbon pricing, high renewable energy targets, energy efficiency, emission limits and other mechanisms.

All could have made the market more efficient and delivered savings to consumers. The latest of these is the proposed shift to 5-minute settlements, a change widely acknowledged as crucial to level the playing field for battery storage, and remove the pricing power ruthlessly exploited by the coal and gas generators.

Like many of the other proposals, it will likely crimp the bottom line of the incumbents. So they are fighting it, keen to push the argument that any impact on their profit margins could have an impact on reliability and supply.

The equivocation over whether we have the tools to manage the energy transition appears to gripped the South Australian government too, whose state is surging past 50 per cent wind and solar and may find itself with two thirds of its demand coming from these two variable sources by the end of next year.

This is perhaps not surprising given the power interruptions of the last year, and the state election that looms next March. The bitter irony is that these events had sweet F.A. to do with the nature of renewables, but of the way the grid has been managed.

The major event cited in the AEMC report was a blackout in South Australia in November 2015, caused by a network fault during repairs to the interconnector to Victoria, and made significantly worse because of how a gas generator responded to frequency and voltage changes.

As the AEMC panel noted, the Torrens B gas generator was expected to reduce output to manage the frequency changes, but did the opposite.

The problem is being blamed on the governor response mechanisms for such plants, an issue raised by numerous analysts and which may be widespread across the country. It adds to concern about the reliability of gas and coal generators that are failing in the heat and at critical junctures in the market.

It might make you wonder why the AEMC and apparently the S.A. government is appearing to put all its eggs in the basket of gas generators, as it appears to have done by insisting only something called “real inertia”, delivered by large spinning turbines, should qualify for its proposed energy security target, at the expense of battery storage.

The draft proposal has stunned the industry. As a report from the CEC highlighted last week, the delivery of inertia can take multiple forms. Citing the same incident in South Australia in November, 2015, Tom Butler wrote:

“Those who advocate for the status quo because of the inertia provided by synchronous generators should be aware that these technologies are far from perfect.

“For example, they can become unstable at low power output. And there is simply no information available on how effectively these generators can respond to fast rates of change of frequency if they started operating before 2007.”

It reminds you of the transition from horse and cart to the automobile. For a while, all cars were required to have a human walk in front of them, waving a red flag, until someone woke up to the folly of the idea.

The hope is that the Finkel Review – due in just over two weeks – might convince more people that we can do without the waving of red flags. The change is upon us and it’s all OK. We just need our regulators and our politicians to catch up.