Ten years after the fact, Kim Warren can still tell you to the minute when the lights went out in Ontario.

It was 4:11 p.m. on Thursday, Aug. 14, 2003, and Warren was the manager of the main control centre for Ontario’s power grid.

He was in the hallway just outside the control room, talking to some co-workers, when the lights flickered.

“It just seemed odd,” he said in an interview. He ducked back into the control room to a scene he won’t forget.

Photo gallery: A look back at the 2003 blackout

“It was disbelief,” said Warren, now chief operating officer of the Independent Electricity System Operator.

“I’ve seen an awful lot of things go wrong. But when I first went in the control room, all the indications of trouble were off the scale.”

How far off? The indicators showed that Ontario was short 8,000 megawatts of power. That’s about one-third of the required supply on a hot summer day.

“Usually anything over 500 megawatts is indicative of trouble,” said Warren. “I don’t think I’d ever seen it over 1,200.”

“This was the entire power system imploding, falling down. That wallboard we have was lit up like a Christmas tree.”

There were six controllers in the centre, and Warren asked the senior controller: “Is this real?”

He replied quietly: “Give me a minute,” just as a call came in to report that the Bruce B nuclear station had disconnected from the grid.

“The senior looked at me and said ‘Yes.’ That’s all he said.”

“I said to him: Let’s implement the restoration plan. I will get you some help.”

“I turned around and reminded myself not to run. It was important to walk, because we have to remain calm. We’re the rudder in these things. I remember walking to the door. Some of the people I’d been talking to were still there.”

“I said: Give me every control room operator who’s here and bring them into the control room. And I said: Don’t let anyone leave the building. Because we didn’t know who we were going to need next, and we didn’t know how long it was going to last.”

(As it turned out, Warren was able to sneak home for an hour at 2 a.m. that night – Friday morning – for a shower and a change of clothes. He didn’t get home again until late Saturday, ending a shift that had effectively started Thursday morning.)

In Toronto – and other major cities around the lakes – civilized chaos was setting in. The blackout knocked out traffic lights and the subway in Toronto. Every intersection was a four-way stop.

Subway travellers tramped home past the stalled traffic, arriving to powerless homes. In many neighbourhoods, spontaneous barbecues erupted as householders emerged from their houses in the dusk.

Fifty million people lost power across Ontario and eight U.S. states.

While most households had power back within a day, businesses across the province faced mandatory or voluntary power restrictions for a week, forcing cutbacks and shutdowns.

A Canada-U.S. task force probing the blackout estimated Ontario workers lost 18.9 million hours of employment, while manufacturing shipments dropped $2.3 billion.

But Warren had more immediate matters on his mind that afternoon – especially when the blackout swiftly turned political.

The blackout had enveloped most of northeastern North America.



New York Mayor Michael Bloomberg and state governor George Pataki almost immediately made public statements that they believed the blackout had started in Canada.

Those accusations quickly flowed into Warren in the control room, who already had his hands full.

“So I had our greatest challenge ever, as far as running the system,” he recalled. “At the same time I’m trying to convince other people that we didn’t cause it.”

The painstakingly detailed report by the U.S.-Canadian task force confirmed what Warren already knew: It wasn’t Ontario’s fault

The blackout’s roots trace back to a string of events in Ohio – some natural, some human.

Trees and hot weather performed the natural factors.

Heat boosted demand for power. That strained generating capacity, and loaded transmission lines – which heated and started dipping lower toward trees that had been allowed to grow up beneath.

Then a unit at the Eastlake coal-fired generating station near Cleveland had a breakdown at 1:31 p.m. – meaning some transmission lines had to work harder to deliver power from outside the area.

At 2:02 p.m., contact between a transmission line and a tree in the Dayton area knocked out the line. It also knocked out the ability of the agency called MISO, which controls the power grid in the area, to properly assess the system for the next three and a half hours.

System operators couldn’t see that with a generator down and a major transmission line out, other lines were overloading, and creating problems for FirstEnergy, the local power company.

First Edison (FE) itself was blind. Its alarm and control systems had failed, but “for over an hour, no one in FE’s control room grasped that their computer systems were not operating properly,” the task force concluded.

From there, the outage went from bad to worse.

Between 3:03 and 3:42, three other major transmission lines sagged into trees and burnt out.

But system operators were unaware of the developing crisis until they started getting urgent phone calls from generating plants and line crews around the region.

Finally another transmission line failed at 4:05, blacking out most of the Cleveland-Akron area.

What had been a local event suddenly mushroomed across northeastern North America. As some circuits failed, power surged onto others, overloading them and tripping off circuit breakers.

Overloaded lines and circuits to the west of Cleveland tripped off. As the task force put it:

“With paths cut from the west, a massive power surge flowed into New York and Ontario in a counter-clockwise flow around Lake Erie.”

More circuits tripped off.

“The entire northeastern U.S. and eastern Ontario then became a large electrical island,” the task force wrote.

Within that island, demand for power exceeded supply. But with no way of drawing in more power, the entire system became unstable and crashed.

That’s what Kim Warren saw at 4.11 p.m.: “I t probably took about 90 seconds to implode.”

By 4:13 p.m., Ontario and most of the lower Great Lakes states were without power.

The blackout was more than an inconvenience.

It underscored some gaping holes in Ontario’s grid operations.

The power system needs power to operate – and emergency back-up supplies turned out to be inadequate.

Loading... Loading... Loading... Loading... Loading... Loading...

At a Toronto area Hydro One control centre, a backup diesel generator failed to kick in, leaving staff in the dark initially as they tried to assess the problems and resurrect the transmission system.

At the Canadian Nuclear Safety Commission in Ottawa, the agency’s emergency operations centre couldn’t function because the building where it’s located had no power, and no back-up supply.

At Ontario Power Generation’s Pickering B nuclear station, another problem surfaced. With the electrical grid down, there wasn’t enough auxiliary power to maintain the pumps in the emergency cooling system. The pumps were out for five and a half hours.

Even where auxiliary power was not an issue, the nuclear plants were in trouble, because with the electricity grid in a state of collapse, there was no place for them to send their power.

The obvious answer is to power them down – but that’s not something that nuclear reactors like to do.

Bruce Power’s plants fared the best. Three of the four were able to throttle back their output without a complete shutdown, then reconnect to the grid within five hours. The fourth needed repairs, which took nine days.

But many of the reactors operated by Ontario Power Generation had to shut down completely. That’s a serious process, and it takes days to get them back up to speed.

One of the four reactors at Darlington was able to get back online within hours, but the three others took three to four days. At Pickering, the final reactor didn’t get back online until Aug. 29 – two weeks after the initial power failure.

The reduction in output from the nuclear plants left Ontario limping, with appeals to industry and consumers to cut back on power use extending for more than a week after the crash.

During that time, Warren and his teams of operators continued to face the delicate task of reconnecting thousands of kilometres of transmission lines, in concert with Hydro One.

It has to be done correctly. Unbalance the system, and everything can collapse again. It’s a task that grid operators are trained for, but had never faced on this scale.

Looking back a decade later, Warren still feels pride in the way Ontario bounced back from the catastrophic outage.

But he still wonders how the system could have gone so terribly wrong elsewhere:

“I guess we were disappointed that neighbouring operations failed, and caused Ontario untold amount of grief,” he said. “Businesses were significantly hurt by this.”

Could it happen again?

It’s not a question that Bruce Campbell, now chief executive of the IESO, likes to answer.

A senior executive at the IESO in 2002, he, too, vividly remembers the night – turning his car around to get back to work just as he started to drive home to help deal with the crisis.

“Never say never,” Campbell says when the question is put.

But he does point out that the blackout led to changes that make a similar crisis less likely.

Part of the problem with the 2003 blackout, Campbell said, was that there were no clearly spelled out and enforced reliability standards for power systems.

Ontario already had enforceable standards – with utilities or power agencies liable to fines if they didn’t make the grade. But in 2003, Ontario was the only jurisdiction in North America that did.

The blackout changed that – probably the most important result of the post mortem, he said.

U.S. jurisdictions have now adopted tougher, enforceable standards, he said:

“There’s been a lot of action since to clarify reliability standards, clarify their application.”

This led to the U.S. following in our footsteps and making the rules mandatory and enforceable.”

Ontario hasn’t stood still, Campbell added. The IESO is installing a new simulator where operators can practise responding to a whole variety of typical and non-typical events.

Other grid operators are also getting simulators.

So do the new standards, and the more rigorous training, mean that it can’t happen again?

Campbell won’t be backed into that corner:

“The enforcement and compliance mechanisms are substantially more robust than they were at that time, so I think the probabilities certainly are down,” he said.

But – as the lengthy blackouts in pockets of the city following July’s floods showed – it still pays to be prepared.