SA Blackout: Three towers, six windfarms and 12 seconds to disaster

Finally, the gritty info we’ve been waiting for: The Australian Energy Market Operator’s (AEMO) preliminary report. The message here is of how a combination of both transmission towers failing and probably the auto-shut-off of wind farms combined in 12 seconds to crash the South Australian system. It’s looking awfully bad for the wind industry. The AEMO pins the crash on the sudden reduction from the wind generators, but stops short of declaring why they dropped power so suddenly. Was it the auto-shut-offs, lightning strikes, a software glitch, turbine failure, or was it a key transmission line that broke? Reneweconomy is about the last-man-standing trying to defend the wind industry in Australia. Giles Parkinson argues it was the third transmission line that took out some wind generation.

Even if the third transmission tower took out two “farms”, the fragility of wind-dominated grids is on display. And above and beyond this, South Australian electricity is a management debacle. The only question is, which mistake was the worst: Is this is epic indulgence of running the wind farms flat out in a storm only to trigger a blackout with their auto shut offs? There’s a compelling case, but there are tenths or less of a second between events in these graphs, and no confirmation.

If it was transmission towers that ultimately broke the system, things don’t look better for wind power which needs so many long transmission lines to capture energy from sites spread far and wide, rather than connecting a few centralized spots like coal stations — and that’s expensive (thanks to Tom Quirk for pointing out that).

We’re still left wondering why were these towers so weak, was it freak tornados — where is that documentation? Then there is the unknowable — could it have been prevented if the Port Augusta coal station was still running, or if the wind farms had turned off earlier in an orderly fashion, or if the transmission towers had been solid?

The bottom line is that wind energy comes at a very high cost and makes the system either very expensive or horribly fragile or both. Given that wind farms aren’t providing cheap electricity — when the infrastructure and the costs of having back up “spinning reserve” and baseload is taken into account — what’s the point of adding all this risk to the system? To change the weather?

How many engineers saw this epic fail coming?



The grim detail: The SA electricity network went from go to woe in 12 seconds

There were faults on three transmission towers over 12 seconds. At the same time there was a sudden “reduction” in output from six windfarms within 6 seconds. Presumably this is when the storm triggered the automatic cutoffs at wind farms, but we can’t say that from this report – it all overlapped. What a mess! So play forensic detective with me and pick through this data.

It appears that the reduction of generation in Hornsdale and Snowtown II wind farms were the killer last straw, but it also appears that the Heywood interconnector to Victoria was already well over the safe limit in the seconds before this. Too much energy was being pulled over into South Australia — perhaps the interconnector was destined to fail anyway. Against that is the diabolical size of the combined Hornsdale–Snowtown crash of 180MW.

The storm rolled in with a north-west to south-east diagonal frontline and must have hit several wind farms and transmission lines at the same time. The dominant form of power in SA at the time was from wind, and mostly from a bunch of “farms” to the north. All up, wind was supplying 880MW out of a total of 1900MW.

This is the twelve seconds that mattered. This is how fast the whole system had to shut down.

(Note that in Grid-land, an open line is a bad thing, it means “no electricity”. A closed line is good, it has no gap where the electrons can’t flow.)

From 4:18 and 15 seconds things unraveled very fast (that’s AEST time).

This graph below shows the leading six minutes of the main Heywood interconnector – it was happily supplying 525MW, but as the wind farms cut out up to 800MW was being pulled out of Victoria (that’s the death-spike!). That was 200MW over the design limits. At 15.8 seconds after 4:18pm the interconnector shut down.

In the table below we see that in the space of 6 seconds, six windfarms suddenly reduced their output. The last two windfarms took out 180MW in two tenths of a second from the system at 4:18 and 15.1 and 15.2 seconds. This reduction is what caused the massive overdraw from Victoria through Heywood, which then shut down six hundredths of a second later. (As pure electro-trivia, we note that briefly Hornsdale and Snowtown were sucking 2MW each from a system about to collapse.)

Here are the locations of these windfarms thanks to Aneroid Energy:

Almost all the ones that matter are lined up in a row, but it was Snowtown II shutting a tenth of a second after Hornsdale that were the straws that broke Electronet’s back.

For the nerds, the detailed timing of the event follows:

The transmission lines were failing first from 4:16pm, and by 4:18pm (and 13 seconds) the third transmission line was out of action. It’s not clear here how much generation power was lost from these transmission lines. (See my adapted grid map below). The AEMO remarks that there was no change in generation with the first tower out, but doesn’t say anything about the second and third.

The Grid Map

Just to see if those transmission lines that fell were critical I tried to mark them on the grid map below. Using the descriptions of “Davenport — Belalie” for example I marked crosses between those two locations (but I can’t pinpoint where the exact break is). The AEMO report (3.1.2) doesn’t say if break number 2 blocked the two windfarms North Brown Hill and Bluff at 4:18:08pm, though the graph below this shows them reducing their output almost immediately after, but apparently still generating something according to the table above. Hornsdale is not marked on this map (as far as I can see). It is close to North Brown Hill but the two “farms” are reported as reducing output 6 seconds apart which suggests that transmission tower fall #2 did not block both (the cross was probably left of where I marked it). We’re not absolutely sure it even blocked one…

Does anyone know where Port Augusta used to fit on this schematic? UPDATE: I’m guessing it would be between Davenport and Cultana and Olympic Dam. ie on the far left main line after the Brinkworth branch).

NOTE: I haven’t marked in all 22 transmission tower faults, so a clear line does not mean a fault had not already occurred in one of the other 4 tower failures before the blackout. (Can anyone figure out where those failures were and if they were fixed before the blackout?)

During the 12 second Interconnector spike

Clearly the 200MW draw on the interconnector is the key moment. So here is a graph of that spike spread out over 12 seconds:

The thing that triggered the beginning of the spike above 600MW was the Davenport–Belalie 275kV line (cross 2 on the schematic grid map), followed in a fraction of a second by four wind farms reducing their output. At this point the interconnector with Victoria is already far over the safe operating limit. Then the Davenport — Mt Lock line breaks, pushing things up higher, followed a tenth of a second later by the last two bigger windfarm reductions, the 180MW pair which pushes the draw from Victoria over the interconnector to nearly 900MW.

Can we blame that transmission tower?

Giles Parkinson at Reneweconomy puts on a brave face and argues that the critical point was the third tower (Davenport – Mt Lock) going out:

…the Coalition will point to the loss of 315MW of wind power highlighted by AEMO in the press release after the collapse of the last of the transmission lines that preceded the failure of the inter-connector. At which point all the remaining gas and wind generators tripped.

But there is a question about whether this loss of wind capacity really mattered. The data in the actual report suggests not.

Wind generators were producing a total of 883MW at the time (gas was providing 330MW and 613MW was coming from Victoria) – and had ridden out the loss of the first two transmission lines.

A small amount of wind capacity dropped out after the second transmission line collapsed, possibly – the operators say – as the result of lightning strikes and a software glitch that has since been rectified.

But as this chart below shows, there was no impact on frequency. [That's Fig 4 at the top of this post -- Jo] It was only the failure of the third transmission line at 1615.18 that some generation was lost, the frequency dropped the system went black 1.2 seconds later.

Not so. The third line went down at 14:18:13. The frequency of the SA grid started falling apart from 4:18 and 15 seconds, exactly when Snowtown and Hornsdale went down. The frequency was smashed when Heywood went down, but the two faults of transmission lines causes jiggles, it was the loss of 180MW that dropped the frequency to 49.25Hz which tripped the interconnector.

The loss of the third transmission line took away the delivery mechanism for two other wind farms, which suggests it wouldn’t have mattered which power source was operating on that line. Within another half a second, all remaining gas and wind plants had gone after the interconnector tripped.

Giles seems to be sure the third tower stopped generation from two wind farms, but it doesn’t say that in the AEMO report. Hornsdale wind farm which is near that break was still generating (for another 2 seconds!) The other two stations close to that break (#2) are Bluff and North Brown Hill, which were already shut down (and for 6 long seconds).

The best friend of renewables in Australia is left to lamely call on what the report doesn’t say.

The report does not say why this happened, or why they stopped generating. It could be because they had nowhere to send their output. Or that, as mentioned earlier, some were hit by lightning, or tripped after repeated voltage drops.

Nor does the report does not say if the total blackout would avoided by having a brown coal generator on line, or if the outcome would have been any different with no wind power.

The report also point to problems with conventional generation, saying that contracted but un-named providers of “black start” services – peaking gas fired and diesel power stations – failed to deliver and could not be used to restart the main gas generator, meaning the operator had to wait until a new link was established with Victoria.

Frequency hell

To give you some idea of how important frequency is and how fast it falls over, here’s a couple of paragraphs from the AEMO report. At the Heywood interconnector, the extreme frequency limits appear to be 4Hz for a “quarter of a second”:

Generator performance standards after 2007 require generating units to remain on line for a Rate of Change of Frequency (RoCoF) of 1 Hz/second for 1 second as a minimum, and up to 4 Hz/second for 0.25 seconds. RoCoF must be maintained within this limit to prevent damage to generating units and effective operation of protection relays and emergency control schemes such as the automatic Under Frequency Load Shedding (UFLS) scheme.

The sudden loss of around 850–900 MW of supply to SA due to the tripping of the Heywood Interconnector resulting in a rapid reduction in the power system frequency.7 AEMO analysis has identified that the RoCoF was between 6 and 7 Hz per second. Consequently, UFLS was not able to arrest the frequency decline and as a result the frequency fell to zero. Note that generating units are unable to operate (and are not required to do so) where frequency is below 47 Hz. With the frequency below 47 Hz, generating units subsequently tripped off line resulting in the SA region Black System.

This is the real story of dire problems with a wind dominated grid. Even if it was transmission towers that crashed the system, and even if the auto-shut-off stupidity could be managed away, that still leaves the grid very fragile because of the frequency dilemma. A stable grid needs “synchronous inertia” — big reliable turbines that drive at near constant speeds. Coal turbines are 600 tons and spin at 3000 rpm. That’s inertia.

What of the other 19 transmission towers?

Of the transmission towers, seven were damaged before the blackout and 14 afterwards.

Graham Lloyd, The Australian

And seven big towers were damaged in the lead up to the blackout.

But AEMO said data currently available indicates that the damage to the Davenport to Brinkworth 275 kV line on which 14 towers were damaged “occurred following the SA Black System”.

The big event was a 123 MW reduction in output from North Brown Hill Wind Farm, Bluff Wind Farm, Hallett Wind Farm and Hallett Hill Wind Farm at 16.18.09.

It was wild weather, but not that extreme

The AEMO report sums up the weather, but as far as weather disasters go, it doesn’t rate. This is not a “Yasi” or even a baby-Yasi.

Throughout the duration of the event, long periods of sustained winds of 50–70 km/h were experienced across South Australia…

Wind gusts, notably more erratic in occurrence and frequency during intense weather systems, were significantly stronger than the reported sustained winds. Peak winds gusts on Wednesday 28 September of 90–110km/h were reported for locations…. including Snowtown…

The system as a whole brought 40–60mm over large parts of southern and south-eastern Australia, exceeding 100 mm in parts of south-eastern South Australia and the interior of Victoria. Large and destructive hail, together with large amounts of small hail were observed with both thunderstorm bands on Wednesday 28 September.

The key weather contribution was this:

In particular, a line of severe thunderstorms moving in a northwest-southeast line from Snowtown to Blanchtown between 15:00 and 17:00 CST would have led to large hail, damaging wind and a tornado near Blyth.

So the stormfront ran diagonally and hit a string of windfarms at the same time. The actual recorded peak wind gusts don’t correlate with the disaster, they were spread out all afternoon, not at the key time of 3:48pm in SA time. It was pretty bad luck that the storm front hit at the right angle to take them all down at once, but Snowtown is not in the same line, or on the same transmission line and would have been hit earlier, and this wind direction is not unusual for the region. Furthermore, with wind farms often clustered along the top of a row of hills, it’s probably not that unusual to find them in a “string” arrangement. With faults and cut offs occurring for several hours, the electricity grid was being peppered with “contingencies” and earlier warnings for months had predicted state-wide blackouts.

h/t to David B, Pat, Scott, Geoff Derrick, David E. David of Cooyal, OriginalSteve, El Gordo, ROM…

BACKGROUND to the SA Electricity crisis (all the links).

People saw The South Australian black out coming. There were warnings that the dominance of renewables made it vulnerable. Then when it came, it all fell over in a few seconds — read the gruesome details of how fast a grid collapses: Three towers, six windfarms and 12 seconds to disaster. Ultimately the 40% renewable SA grid is crippled by complexity. The AEMO Report blames renewables: The SA Blackout was due to lack of “synchronous inertia”. The early estimates suggest the blackout costs South Australia at least $367m, plus their normal electricity is twice the price, and there are reserve shortfalls coming in January 2018 (pray for a cool summer). Welcome to the future of unreliable electricity: Rolling blackouts ordered in SA in 40C heat. And more bad luck for South Australia, yet another blackout, 300 powerlines down, 125,000 homes cut off. See all the posts on Renewable Energy and Electricity Grids.

VN:F [1.9.22_1171]

please wait... Rating: 9.5/10 (123 votes cast)