It’s a typically cold July evening in Melbourne, and a day after planned network outages in Victoria and South Australia, the wholesale spot price for electricity is spiking. Somewhere in the country, an energy generator is cashing in on market volatility.

Two of Victorian energy minister Lily D’Ambrosio’s advisers are hunched over their mobile phones, eyes glued to the PocketNEM app. This app has become an obsession for energy insiders as Australia’s power price wars play out. It gives traders a real-time state-by-state view of what’s happening in the National Electricity Market (NEM), including which states are importing energy at peak prices. For the regular consumer, it’s a fascinating insight into the underlying volatility in Australia’s wholesale energy market.

Politicians, including federal energy minister Angus Taylor, generally talk about the “ripoffs” at the retail end of the market. On Wednesday, Taylor called out the big energy companies for “hiking their bids” and “price gouging” after the closure of big generators, such as Victoria’s Hazelwood Power Station. But the pricing journey for the energy you eventually buy starts in this spot market – the one fluctuating wildly on the PocketNEM app – something few outside the energy business ever talk about.

Prices on the wholesale spot market can range from minus $1000 a megawatt hour to more than $14,000. Of late, the average price per megawatt hour in Victoria has been close to $100, up from $57 just five years ago. As with any trading market, the engine is greed.

The government is currently pushing a plan to effectively cap retail prices, a move that would require legislation. The energy retailers blame poor policy for price rises and have suggested price gouging is a myth.

The biggest problem, says Energy Consumers Australia chief executive officer Rosemary Sinclair, was identified by Australian Competition and Consumer Commission chair Rod Sims during this year’s inquiry into electricity pricing.

“He uncovered a very important cultural problem in the sector, which is that nobody, neither industry or regulators, were taking accountability for the price outcome.

“Everybody just was saying: well this is the price of the network, this is the price of the wholesale energy, this is the price of the green schemes, this is the price of retail competition, so it all gets passed through to consumers. And when wholesale energy prices became very volatile a couple of years ago, retailers just passed them on in full, thinking that was all fine and dandy.”

One of the most important agencies in the energy market is the Australian Energy Market Operator (AEMO). Its concern isn’t prices, it’s energy security – making sure the lights don’t go out. For Australia’s energy system to work, to avoid power outages, supply needs to match demand perfectly. If supply is forecast to fall below demand, AEMO steps in and either uses its directions powers to force supply; asks power companies to start switching off supply to some customers (known as load shedding); or uses back-up resources it has specially procured to deal with an expected peak.

Under the watchful eye of AEMO, energy generators bid into a stock exchange at five minute intervals, with prices settled every 30 minutes. This is where things get interesting.

Under the watchful eye of AEMO, energy generators bid into a stock exchange at five minute intervals, with prices settled every 30 minutes. This is where things get interesting. Energy generators can rebid in their power right up to 67 seconds before it’s dispatched into the grid, giving them plenty of time to shift their output to high-price periods to increase earnings. Energy generators are expected to bid “in good faith” but researchers at public policy think tank the Grattan Institute say many have still been able to “game the system”.

Guy Dundas, an energy fellow at the Grattan Institute, posits this scenario: a coal-fired generator with spare capacity can ramp up and dispatch more capacity at a low price. “The dispatch engine will say the price is low, it looks like they’ve got this covered. What the gas turbines will read out of that is ‘I’m not needed, it’s not economic for me to run,’ ” says Dundas.

Once the gas turbines power down, the coal-fired generator can then rebid at a much higher price because it’s effectively the only one that can ramp up quickly and meet the demand. Gas turbines take some time to synchronise.

Given the high volatility in the wholesale energy market, retailers rely on futures and hedge contracts to smooth out the wholesale price they pay for energy. Some say it’s here, in the energy futures market, that regulators should be focusing their attention.

Adrian Merrick, who left his job leading EnergyAustralia’s retail business to set up his own electricity retailer, Energy Locals, says when the regulators investigate the market they focus on spot markets. Here there’s been only a few instances of generators being caught out.

“The AEMC [Australian Energy Market Commission] came out and said, ‘We haven’t found any evidence of gaming the market, unless there is too much concentration in the generation market, in which case it’s a problem.’ Well, there is concentration in the generation market!”

Merrick says the gaming he sees happening at the moment is most rife in the futures market.

“I know for a fact that one of the big generators in Victoria is sitting on top of a heap of contracts for Q1 [quarter one] that it just isn’t selling. They could fundamentally bring down the forward prices, but they’re choosing not to, they’re just withholding those contracts from the market, so that sort of thing is pretty dirty.”

This week, Australia’s energy companies came out swinging against the federal government’s current legislative agenda to get power prices down. The government, in one of the few areas of bipartisanship in energy policy at a federal level, wants to force price caps, which it says are in line with the “default offer” recommended by the ACCC in July.

The Australian Energy Council, which lobbies for 22 major power companies, on Tuesday circulated legal advice saying the enforcement powers mooted could breach Australia’s constitution. Angus Taylor responded by saying the large generators were exploiting customers. “Seventy per cent of the market controlled by three players have done the wrong thing,” he said. He told the ABC’s RN Breakfast, “we get legal advice on these issues and we’re confident in our position on this issue as with other legislation that is in that package”.

Rosemary Sinclair says the ACCC’s report on electricity pricing has been “unhelpfully politicised”, there’s been no proper process to consult on its recommendations and refine them and talk of default offers just adds to the confusion for consumers.

She says in the past three to four years, the standing offer prices the so-called default offer would replace have morphed from being a safety net type product to being used as the base from which to offer discounts. “It allows retailers to say, ‘I can offer you 40 per cent discount’, so it’s being used as a marketing ploy.”

Sinclair says rather than be forced to fight price caps, the retailers should take their standing offer price back to what it was a couple of years ago. That, combined with the reference rate the retailers have already agreed to pursue, would allow people to better compare offers and the ACCC to monitor how much better consumers are doing.

“You can go into the grocery store now and compare unit prices and that enables strong competition and good choices. In mobile phones you can say, ‘Okay, this will cost me $1800 over two years, versus this other plan of $2200 over two years’, and the process of choosing is really easy so people aren’t frightened.”

EnergyAustralia chief customer officer Chris Ryan says less than 12 per cent of the company’s customers nationally are on standing offers, but it’s costing those who are a lot of money.

“We’ve written to all of them over the past year encouraging them to move to better deals, but they’re getting harder to budge. Under current laws, retailers are prohibited from transferring customers on default contracts to cheaper market offers without their consent.”

Sinclair says that in Victoria the 7 per cent of people on standing offers are contributing 18 per cent of the revenue of the big retailers, so there’s a very clear inequality problem.

Energy Locals’ Merrick says caps are a good idea because they would reduce costs for customers who are currently funding discounting by retailers that he says is “blatantly anti-competitive”. But he’s worried the generators could use the opportunity to squeeze small retailers out of the market, by keeping their retail prices low and making money in the wholesale market. “They’d have to sacrifice a couple of years with lower margins … but retail prices would be low, and it would force other retailers to sit on or above the cap, meaning they would go out of business.”

Some, including the Greens, are calling for a renationalisation of the energy grid.

“Unless you want a tax bill that’s phenomenal, it’s not going to happen,” says Gavin Dufty, manager of policy and research with St Vincent de Paul.

“The states own more assets than the private investors now. Government’s got skin in the game, too, they can borrow at a cheaper rate and then they use it as a revenue stream, which means they don’t have to tax. Well, they tax inadvertently through people’s energy bills. Which is really nice, for them.”

The government is also threatening to break up the big energy companies, but few think this will happen.

“People are already starting to lobby the probable energy minister over the existing energy minister,” says Merrick. “I don’t think they’re too bothered by that threat … it would just be a huge shit fight.”

This is the first in a series of articles explaining Australia’s energy policy mess. Next week, as Victorians go to the polls, we explore whether the state’s renewables-focused energy policy will make for blackouts in the summer ahead.