



Image courtesy of Mario Sanchez, AP

No, Beyoncé’s revival of Destiny's Child did not cause the blackout during this year’s Super Bowl game on February 3, 2013. It was combination of technical and communication failures that contributed to the partial power outage according to an independent analysis.

Dr. John Palmer of Palmer Engineering & Forensics did an investigation and found that a recently installed relay had a malfunction or “misoperation” that caused it to trip in an unpredictable way. The device's trip level had been left at the factory default setting, which was inappropriate for its application in the dome. Dr. Palmer was hired by Entergy, which supplies electricity to the Superdome, and is the stadium's management company. Palmer's report also cited “inadequate communication between the manufacturer and the utility” as a contributing factor. This electrical relay device was installed by Entergy specifically to prevent a power failure at the dome.

The report also notes the relay had a design defect, and under testing it did not perform entirely as its instruction manual said it was supposed to. It says the factory default setting of the relay was inappropriate. The device has been permanently removed from service.

The cause of the blackout may have stemmed from a project beginning in 2011 in order to protect the power by ensuring the protection of a cable linking from the power grid to the Superdome for the Super Bowl.

The relay was created by Chicago-based company S&C Electric company. The vice president of strategic solutions of the company, Michael J.S. Edmonds, told the Associated Press that “if higher settings had been applied, the equipment would not have disconnected the power.” After this comment, tests were conducted on manufacturing of the product and showed that while one of the relays to the dome worked, the other failed.

This is all well and good, but good engineering design must allow for every possible fault. This is, of course, limited by how much you are willing to spend to ensure the least likelihood of any failure.

Scenarios for success

Let’s look at possible scenarios for a more robust system, through the eyes of a “Monday morning quarterback”—me (pun intended as usual!)

M.J. Kornblit, P.E., Sr. Systems Engineer, GE Beijing 2008 Olympic Games Solutions wrote a white paper entitled “Power distribution systems for the Beijing 2008 Olympic games”

Kornblit says that Field-of-play (FOP) lighting is typically a very large imperative load and every aspect of its design, including the power supply must receive special attention. Most FOP lighting relies on metal-halide technology even a minimal voltage disturbance can cause an extended outage.

The power supply must be fully redundant, backed up by generation, and the lighting divided into two equal groupings. During competition each group must be isolated from the other and fed from a separate source, each source capable of feeding the full load if necessary. It is also necessary to physically interlace the lamps between groups to minimize the overall impact on the field if an outage does occur. A random interlacing of the three phases fed to the lamps is also required to reduce possible visual flicker, observable during certain conditions.

Some FOP lighting designs incorporate large UPS or similar systems to make certain the highest level of power quality is retained throughout the critical need period.

Simon Liang, AE Manager, GE Greater China and M. Kornblit, GE Olympic Games Solutions wrote another white paper “Paralleling Temporary Power at the Beijing 2008 Olympic Games”

Here is what they suggest: The IBC electrical power distribution system consisted of numerous 400V switchgear lineups distributed throughout the facility, the configuration of each is as illustrated in Figure 1 below2 . The typical main bus scheme provides multi-power sources with backup as sketched. Under normal conditions, the system is powered by main source S1.

If the main source is lost, then S1’s main breaker opens and the tie breaker automatically closes, connecting backup power S2 to the bus. At the same time, because an abnormal condition exists, the backup emergency generators start-up and serve as backup to S2. The number of actual generators (one, two or three) in operation to serve the bus varies between locations and depends on critical load needs. Wherever multiple generators are provided they are always synched to each other on their common bus prior to possible use. Should source S2 be lost, the generator(s) will provide emergency power to the system. All power sources are intended to run independently, only one source provides power to the system at any given time under the automatic switching conditions described and all critical control is through PLC (Programmable Logic Controller).

Image courtesy of white paper2

Power loss, unfortunately, does occur on the bus during these emergency-operating conditions. The outage time per switching cycle, including sensing, signal transmissions, and internal trip times, is roughly ten to one hundred milliseconds. Even short disturbances such as these can result in damage and extended outages to these most sensitive critical loads, and impose long restart cycles for their (and other equipment) internal control processes. UPSs were therefore applied wherever possible to assure continuity of high quality power to the most critical loads.

The reliability and value of the fundamental power source remained of great concern in such a high visibility application for confident operation under any eventuality including bypass modes. It was agreed to minimize such outages wherever possible by incorporating a transitional parallel operation mode between sources. That is, paralleling would be permitted, for this application only, and for periods not to exceed a maximum defined length of time. These possibilities existed during power recovery switching cycles: from backup operation mode, returning to the main source, and during test and scheduled maintenance operations.

This agreement was far from simple since paralleling the customer’s power system with that of the Power Bureau constituted a change to normal practices and related standards, considerable safety and operational related concerns, and a venture into the potentially risky unknown. Many detailed discussions were conducted between all involved parties over a period of two years.

Final agreement occurred several months prior to the actual beginning of the games, and after the installation and startup of the original non-paralleling switchgear had occurred. Revision of the already energized switchgear was carefully planned, implemented in the field with highly reliable factory constructed components and subsystems, and successfully tested under strict scrutiny.

The Paralleling Transfer Scheme2

Parallel operation of supply sources within a power system has been shown by many studies to provide added reliability to the overall operation of the power system and may be necessary for certain critical applications. Such operation adds system complexity and requires suitable planning, appropriate system protection and control, and critical evaluation of the system’s needs and capabilities. When done properly, however, it is possible to achieve even when the system equipment has already been installed and placed in service. In all cases proper consultations and approvals are necessary.

The IBC was a large facility with many LV substations, the basic schematic diagram for each is as shown in Figure 1 above2 . A more detailed diagram of the final revised system configuration is shown in Figure 2 below2 . A number of items had to be clearly considered to make this project a success:

Image courtesy of white paper2

Controls and Communications2

To ensure the reliability, and fast-response time of the control system to the changes and needs of the LV and generator systems, careful consideration was given to the selection of system hardware and communication style. The Multilin F650 protection relay was specified for its wide range of protection and communication functionality as well as control logic and setting flexibility. The GE Fanuc Pac RX3i PLC was chosen to provide all the logic, I/O and fast interface capabilities needed for the system paralleling transfer, and to properly communicate with the generators and their independent PLCs, the Multilin F650 and all breaker controls. Selecting these two key items minimized the number of other control devices typically required for such applications since the F650 (see Figure 3 below) incorporates reverse power and synch-check functions as part of its capabilities in addition to all the metering, protection and control necessary to sense outages, determine load levels, communicate to the PLC’s, and protect the circuits regardless of the direction of power flow. Likewise, the RX3i’s high performance 300 MHz microprocessor-based controllers, 10 Mbytes of user memory, universal programming environment capabilities, and its ability to cross multiple hardware platforms, assured everyone the fast response and high reliability needs of this project could be met.

Redundant control power including UPSs was incorporated to supply all critical controls in the circuitry. Because of the large distances separating the LV switchgear from their supporting temporary generator sources, redundant fiber optic communication cables were used here to minimize the need for auxiliary supporting hardware, decrease possible induced interference and assure overall communication reliability between these two isolated but critical parts of the system.

Protection Devices2

Ina conventional power distribution system, each circuit breaker is required to have its normal complement of protection functions including long time for overloads, instantaneous for short circuits, and others, depending on the application, which can include short-time and GF to assist in coordination and earthing protection. When power sources are paralleled, additional protection must be added at various key positions in the system. This protection includes a means of confirming that the voltage magnitude, phase angles, and frequencies between the two systems are within a safe tolerance level. Directional sensing capabilities must also be included in order to limit the flow of power in undesirable directions, and to assure proper coordination in both directions, since the protective settings are usually different in each direction.

The protective functions included within the trip unit of a standard low voltage circuit breaker cannot typically differentiate between the directions of power flow. For this reason its overcurrent settings will trip at the same magnitude whether the load is flowing upstream or downstream. The F650 relays were added to obtain this important capability as well as the many other protection, sensing and communication functions required for this application.

Logic of Paralleling Transfer2

The logic for parallel operation must be carefully considered and flow charts prepared. Each breaker must operate as part of a predefined sequence, system conditions and permissives confirmed, and each alternate power source sequenced into its proper standby or primary position in the system depending on its state of operation, the previous and existing conditions. The substation PLC served as the central controller for the main switchgear, gathered all local data, performed all necessary critical control, communicated all interfacing with the standby generators and their controllers, and confirmed all actions. The PLC can also perform load-shedding functions if required and requested.

For this application normal power is supplied by main source S1, the backup power (S2) is in hot standby, and the gensets are under cool-standby. When the main source power is lost, the PLC commands the S1 main breaker to open. Once confirmed and following verification of a sound bus and availability of the alternate source, a close command is sent to close the S2 tie breaker. Power is automatically transferred to the backup source with minimal delay, and the gensets are commanded to start via communications. Within about 15 seconds the genset are available as the hot-standby. Alternative logic must also be provided to assure proper operation and immediate notification if any of these planned operations fail to perform properly.

Separate sequences must also be considered and included for the recovery cycle to permit a return to normal power conditions. For this system, different sequences and methods are incorporated depending on which source is supplying the system at the time of recovery. The process may include the paralleling of either both utility sources or the gensets with the utility source. In either case, verification of safe transfer conditions is confirmed through the protective relaying and sensing, and the period of parallel operation is transitional.

Withstand Verification2

A major concern for parallel transfer schemes is that the fault current capacity of the system, which dramatically increases during the period of parallel operation, may exceed the short-circuit ratings of the equipment being paralleled. Appropriate system modeling and evaluations must be conducted to confirm these ratings and their duties are compatible.

Such evaluations were conducted for this application, investigating maximum short current fault conditions of various types and differing system worst-case fault conditions. Below in Figure 4 is an example of one of the case conditions prepared for the IBC showing only the main LV switchgear. Here three 1250kVA generators are run in parallel and connected to the main switchgear bus. The results show that the maximum three-phase short current can reach 54kA under these conditions, well below these equipment ratings.

Metal Halide vs. LED lighting3,4,5

Metal Halide lamps generate light from an arc formed between two electrodes. They have high luminous efficacy (almost 100 lm/W)

Metal Halide lamps need an igniter of some sort. A “MHz superimposed-pulse igniter” is widely used and operates with high-frequency pulses (approx.. 1 to 10 MHz), and are suitable for ignition at any stage from hot to cold.

Figure 5: Circuit diagram for a MHz superimposed-pulse igniter (Courtesy of Sylvania)

The ignition coil is usually connected in series with the lamp. As shown Figure 5. The pulse generator generates voltage pulses. The superimposing transformer is to transform the voltages generated in the pulse generator to the surge voltage required to ignite the lamp and to superimpose this surge voltage on the low-frequency lamp supply voltage.

A symmetrical output as shown has the advantage of the fact that only half the voltage is applied at the outputs with respect to ground, which makes it easier to provide adequate insulation inside the fixture.

Due to their “current/voltage characteristic”, electrical (discharge) arcs take as much current as possible from the power supply. To ensure safe operation, a current limiter is needed. There are three options for the current limiter:

A series ohmic current limiter or resistor. In light of the very high electrical losses this is usually not used in practice A series magnetic current limiter or inductor (choke or magnetic ballast) which acts as an almost loss-free “current brake” A series electronic current limiter or electronic control gear (ECG). This is more efficient than the inductor. See Figure 5.

Circuit diagram for the electronic control gear limiter (Image courtesy of Sylvania)

Startup behavior is one of the drawbacks for this type of light. The power outage at the Super Bowl causing the lights to go through a 34-minute recycle to warm up which delayed the game could have been completely avoided using modern LED lighting technology.

A delay with the lights like those used at the Super Dome is that the lamp itself contains gas inside, metal halides. To turn on the light, an arc, much like a welding arc, is created that excites the gas and causes it to shine bright light. The lamps become very hot, hundreds of degrees, when operating. This creates tremendous pressure inside the lamp and due to the pressure, the arc will not reignite the gas until the light cools down and the pressure subsides. This typically takes 20 minutes or more.

LED lighting for stadiums

LED lighting has reached the stage of technology that enables it to be implemented even in field and stadium lighting venues. This technology is another possible help in reducing down time in failures like that in New Orleans at the Super Bowl this year.

Check out this video using HiViz lights with CREE LEDs for fields or stadiums with “instant-on” and “instant-off” capability.

If proper engineered design is put in place and LED lights are used, these are the top two solutions that might have avoided the problem of ‘blackout” on February 3 at the 2013 Super Bowl.

Please give us your expert insights into other possible solutions or comments on this electrical design problem.

References

1M.J. Kornblit, P.E., Sr. Systems Engineer, GE Beijing 2008 Olympic Games Solutions “Power distribution systems for the Beijing 2008 Olympic games”

2 Simon Liang, AE Manager, GE Greater China and M. Kornblit, GE Olympic Games Solutions “Paralleling Temporary Power at the Beijing 2008 Olympic Games”

3 HiViz LED lighting using CREE LED’s

4 Synergy LED lights

5 Sylvania Metal Halide lamps