Venezuela’s melt down: Blackout day six and the grid struggles to reboot

Venezuela has 31 million people and has had almost no electricity for six days. There are estimates on Twitter (#sinluz) suggesting that about half is back up as I write, but the stories of chaos, death and disaster are surely just starting to come out as communication lines open. The water coming out of taps is black, possibly contaminated with oil (are those shots fake?), the Pepsi plant was stripped bare, see the video. People are desperate: Shop owners are apparently shooting looters. At least one tweeter claims people are eating zoo animals. One baker took his own life after his shop was overrun and everything was stolen. It may not be over yet either — the grid recovered almost as far a few days ago, then collapsed again. Indeed, today explosions have been reported at an electrical substation at La Tiama, as well as other substations. What a debacle, a disaster. Babies in hospitals are being ventilated by hand. Many life support systems are off.

Netblocks tracks connectivity in Venezuala, which seems to be a reasonable proxy for power, and clearly electricity is being rebuilt partially, then collapsing again.

Update: 119 hours after the onset of nationwide power outages #Venezuela‘s connectivity is up to 63%, marking significant progress in the restoration of utilities #SinLuz #12Mar ⬇️https://t.co/8pljYDEYae pic.twitter.com/zcBeVkqnKG — NetBlocks.org (@netblocks) March 12, 2019

On the BBC one journalist describes the situation as like living the apocalypse. “They never thought it would come to this”. And he is not even referring to the armed pro Maduro regime gangs that are allegedly arriving on motorbikes and firing on protestors. See @MarcoRubio In the video the crowd is screaming and running for their lives.

Venezuela shows how fragile (and marvellous) a working grid is

History books will be written about this crisis. Matias Delacroix points out in Wired, that it’s very difficult to restart a decentralized grid which has been badly managed and poorly maintained. No one knows exactly what went wrong to bring it down, but fingers are pointing at the huge Guri Hydro plant, which provides a whopping 80% of the electricity.

Normally a blackstart begins with a small diesel unit to kick over a bigger turbine. Then the whole grid is gradually rebuilt bit by bit by “bootstrapping”. At all times the supply has to match the demand, so before an engineer throws the switch someone has to have already done the numbers to make sure things will stay balanced. As each new generator-load segment is added it must also match the frequency and phase set up by the original first generator. It would be dangerous to set up a bunch of little separate grids and then try to meld them together. Getting the frequency or phase wrong can cause explosions. Hence the grid must be built from one point out, and carefully. Wind or solar generators can’t be used in a black start, but a large Hydro plant should be able to do that, assuming people can open the water gates, but restarting a badly managed grid means there are many fragile failure points that don’t reboot properly. After days of darkness it must also be difficult to estimate how much load will be waiting to take new power.

Matias Delacroix, Wired

While distributed systems don’t have a single point of generation failure, they can be more difficult to black start if they do go down, since more generation sites need to be bootstrapped and there are more loads to balance.

Regardless of the setup, the crucial component of all black starts is understanding what caused the outage, having the ability to fix it, and working with a system that can handle the power surges and fluctuations involved in bringing power back online. Without all of these elements in place, says Tim Yardley, a senior researcher at the University of Illinois focused on industrial control crisis simulations, black starts can be prohibitively difficult to execute.

“Reenergizing a grid in some ways is more of a shock to the system than it operating in its norm,” Yardley says. “If infrastructure is aging, and there’s a lack of maintenance and repairs, as you try to turn it back on and try to balance the loads you may have stuff that’s not going to come back up, infrastructure that’s been physically damaged or that was in such a bad state of repair that reenergizing it causes other problems.”

I asked a network engineer once what happens to the energy if, say a large turbine, was connected to a grid at the wrong frequency. Energy can’t just disappear, and if the phase or the frequency flatlined, and and opposing waves cancelled each other out, where would all the joules go? He replied that it could be catastrophic — those kind of events shear through the steel turbine shaft (and these turbines can weigh up to 600 tons). When there are megawatts of energy at stake, mistakes are violent.

Crews attempting to deal with black-starting a frail and brittle grid also face major safety considerations, like explosions. “You have a maintenance issue and a manpower issue, because it’s extremely dangerous to reenergize a system if you have gear that hasn’t been maintained well,” Yardley notes.

Venezuela has faced years of power instability since about 2009, including two major blackouts in 2013 and a power and water crisis in 2016. At times the blackouts were caused in part by weather conditions like El Niño, but overall they have established a pattern of poor planning, mismanagement, and lack of investment on the part of the government.

Yay. Socialism.

UPDATE: China is offering to help get the grid running.

UPDATE: This may take months

Brilliant comment from Lance #5 explaining why this is a nightmare for grid managers and may take 3 – 6 months to sort out. Or longer if they blow too many substation transformers which are custom made.

Lance March 14, 2019 at 4:45 am · Reply

Performing a “black start” is no simple matter. The primary problem is that everything that was connected to the grid when it crashed is still connected. So essentially, the Load is a “Dead Short” from the perspective of Generation. Every inductive load (induction motors) takes 6 times the normal running current to start each and every one. In terms of real and imaginary (complex) power components, the Load appears to be almost purely inductive with a Real component vector of nearly zero. Essentially, Generation must provide 6 times the power it was providing when the grid failed and that reserve simply doesn’t exist. So energizing a substation is an explosive event. The safest / only way to restart the grid is to isolate all of the loads except residential loads and bring up the lower voltage substations (10 kV) gradually in a controlled fashion. The residential load has resistive components ( water heaters, clothes dryers, cooking ovens, etc ) that help reduce the inductive component and provide a unity power factor component to the apparent load. Only after the lower voltage grid is stabilized can the higher voltage transmission lines and substations (110 kV to 750 + kV)be re-energized. Even so, it is a precarious dance of balancing generated power with apparent power. When the generator is connected to the load, it “sees” a reflected wave coming back to the generator that trips the overload safeties and causes the turbine/alternator to disconnect if the apparent power exceeds safe limits. If those safeties aren’t functional, the risk is an exploding substation, alternator, sheared turbine shaft, etc. This is a nightmare scenario. No sane person ever wants to “smoke test” a power grid by trying a black start. The ramifications are frightening. This is specifically why keeping a stable grid operational is a lot smarter than trying to roll the dice with intermittent generation and sudden changes in loads. Greens ought to give the situation in Venezuela a very serious consideration before destabilizing the existing grid in any location on earth. My guess is it will take 3 to 6 months to restart the grid in Venezuela, even if things go swimmingly. If a few substations and alternators are blown out, it could take 2 years. Longer if some turbines are damaged. Substation transformers are custom made to order. They do not exist “in stock on hand” at the power levels needed on a national grid scale. Unit substations might be available in smaller sizes, say 50 to 100 MW. But the high voltage and higher power switchgear and transformers can be a 1 to 2 year lead time item even if you have the cash to pay for them. This is a teachable moment. Smart people will pause and reflect upon what is happening, lest it happen elsewhere. This is not a game sane people want to play. Societies melt down in a matter of days to weeks without electric power, water, food, transportation, communication, etc. We’ve yet to see how bad this is going to get. It will get a LOT worse before it gets better. ——————————— UPDATE: This link to Monash IP Observatory is monitoring live net connections in each district. Many appear to be back up in the normal range of activity. Urdeneta, Sucre is not. Bolivar appears to show blackouts two weeks ago in the lead up. Valencia has some wild spike of activity.

VN:F [1.9.22_1171]

please wait... Rating: 9.8/10 (80 votes cast)