In 2009, project manager Mike Wharton inspected a recently installed steam generator within the San Onofre nuclear power plant’s Unit 2 generator. — Charlie Neuman / Union-Tribune

When alloy tubing in one of the new steam generators at San Onofre leaked a small amount of radiation four years ago this week, engineers at Southern California Edison immediately instituted emergency protocols and shut down the nuclear plant.

Neither of the twin domed reactors on the north San Diego County coast have produced a spark of electricity since.

No one disputes what caused the failure — excessive wear in hundreds of tubes designed to drive hot steam through massive turbines is the confirmed culprit, numerous investigators and analysts found.

But what has become increasingly disputed since the plant went dark is the question of who is responsible for flawed replacement steam generators being installed and who should pay for the failure.

Edison, the San Onofre operator and majority owner, said it had no knowledge of design flaws that led to the Jan. 31, 2012, breakdown. Edison places the blame with Mitsubishi Heavy Industries, the Japanese firm hired to design and build the replacement steam generators.

“SCE was unaware of the steam generator defects until they were discovered after the tube leak in 2012,” spokeswoman Maureen Brown said in a statement. “It was up to MHI, as the designer and manufacturer, to decide what design features to include that would result in safe RSGs” or replacement steam generators.

Billions of dollars are at stake in the plant’s failure, and so far, the lion’s share of the tab is being covered by the ratepaying public.

Following a November 2014 vote by the California Public Utilities Commission, customers of Edison and minority owner San Diego Gas & Electric have been paying $3.3 billion of the $4.7 billion in identified closure costs, or 70 percent.

That balance remains controversial, as numerous lawsuits wind their way through various courts and the commission itself is the subject of state and federal criminal investigations over its ties to utility companies, which own the plant.

GRAPHIC: steam generator tube wear Click for more information.

Both of the consumer groups that negotiated the deal have since withdrawn support for the agreement, citing revelations of undisclosed meetings and backchannel communications between utility executives and state regulators after the shutdown.

At the same time, a number of legal actions and investigations have revealed documents that shed light on the $680 million steam generator project that ended so badly. A key question regarding the ill-fated project at San Onofre is whether the new engines were significantly changed from the old ones.

Edison made no secret that its plan was to make certain changes to the generators, but not so many that it would bring the additional cost and delay of a federal license amendment from the Nuclear Regulatory Commission . That approach brought the project online sooner, but may have caused a missed opportunity to scrutinize a project that turned out to be fatally flawed.

“From Edison to the NRC to the PUC, the system clearly didn’t work,” said John Geesman, an attorney for the Alliance for Nuclear Responsibility, a San Luis Obispo consumer group that is fighting to overturn the San Onofre settlement. “And these documents indicate what Edison knew in advance of the leak, what Mitsubishi knew in advance of the leak, what was concealed from the NRC in advance of the leak.”

Daniel Hirsch, who runs the Program on Environmental and Nuclear Policy at the University of California Santa Cruz, said there is a lesson to be learned from any oversights or missteps: There are huge public risks if regulators become too cozy with companies they oversee — especially in the nuclear arena.

“Fortunately San Onofre was permanently closed before a major accident occurred,” Hirsch said. “NRC estimated a meltdown could result in as many as 130,000 immediate deaths, 300,000 cancers and 600,000 genetic defects. We dodged an awfully big radioactive bullet.”

Improved generator

San Onofre’s steam generators were installed in 1983 and 1984. By the end of their third decade of service, their pipes were getting plugged up, and the units needed replacing.

In July 2010, the replacement steam generator installation was all but completed and the equipment was close to being fully activated. Two Edison engineers, James K. Chan and David J. Calhoun, presented a white paper that same month during a special conference of the American Society of Mechanical Engineers in Bellevue, Washington.

Their nine-page report detailed the various improvements in the RSGs.

In addition to new alloy tubing, Chan and Calhoun identified several design changes they said contributed to an improved steam generator. They singled out the higher number of tubes — 377 more in each of the new generators — and their thinner wall thickness, which dropped from 0.048 inches to 0.0429.

“With (the) larger number of tubing and taller tube bundles, the RSG nominal heat transfer surface area is 116,100 sq. ft., which (is) larger than the OSG (original steam generator) surface area of approximately 105,000 sq. ft.,” the engineers reported.

A larger heat transfer surface area can help produce additional power.

Improvements were also laid out in an article in January 2012 — the same month as the radiation leak — by Edison engineer Boguslaw Olech and Tomoyuki Inoue of Mitsubishi Heavy Industries for the prestigious trade publication Nuclear Engineering International.

The title: “Improving like-for-like RSGs.”

The four-page report lays out many of the design challenges and successes the engineers encountered as they drew up plans for the replacement steam generators at the San Onofre Nuclear Generating Station.

“At SONGS, the major premise of the steam generator replacement project was that it would be implemented under the 10 CFR 50.59 rule, that is, without prior approval from the U.S. Nuclear Regulatory Commission,” they wrote. “To achieve this goal, the RSGs were to be designed as ‘in-kind’ replacement.”

Although the replacement was to be “in-kind” in terms of form, fit and function within the power plant, the team wanted to make “all possible improvements” to the steam generator itself, in the hopes of boosting the plant’s “longevity, reliability, performance and maintainability.”

Critics say a drive to improve the generator design while avoiding more intensive federal review was a profit-driven exercise that proved reckless.

“It was all about greed, and expediency, and they risked lives in the process,” said Charles Langley, a longtime San Diego consumer advocate. “They would have lost millions and millions of dollars in revenue by going through a license amendment process. It would have delayed the deployment of the steam generators for years.”

The company says safety is its priority, and that it wanted to avoid federal review because that step is only required if there are adverse safety implications to the project at hand.

“The 50.59 process permits changes to plant equipment without a license amendment where the changes satisfy certain criteria, for example, where the changes do not adversely affect the safety function of components,” Brown said. “SCE’s goal was to obtain RSGs that would remain within the bounds of 50.59, in order to help ensure that design changes would not adversely affect the safety of the plant. In fact, SCE made clear both in the contract and in the design review process that its overriding priority was to ensure safety.”

Root cause

Mitsubishi Heavy Industries kept careful track of the San Onofre steam generator project. It was, after all, a $680 million undertaking that required extensive discussion and documentation.

In October 2012, nine months after the failure, Mitsubishi delivered a lengthy internal report to federal regulators detailing what caused the breakdown.

A heavily redacted version of the so-called Root Cause Analysis was released in March 2013. It discussed high “void fraction,” a gas-to-liquid measurement that — when elevated — can contribute to the very kind of tube wear that caused San Onofre’s radiation leak.

The report discusses the actions of the design team for active-vibration bars or AVBs, designed to protect tubes from excessive wear.

“The AVB Design team recognized that the design for the SONGS RSG resulted in higher steam quality (void fraction) than previous designs and had considered making changes to the design to reduce the void fraction … But each of the considered changes had unacceptable consequences and the AVB Design Team agreed not to implement them.”

Then this: “Among the difficulties associated with the potential changes was the possibility that making them could impede the ability to justify the RSG design under the provisions of 10 CFR 50.59,” or without the added NRC review.

The design team in question was a joint Edison-Mitsubishi effort, and federal nuclear regulators cited both parties for failures leading up to the San Onofre leak.

As the NRC put it, the problem was “the failure to verify the adequacy of the thermal-hydraulic and flow-induced vibration design of the San Onofre replacement steam generators, resulting in excessive and unexpected steam generator tube wear.”

Edison officials say they relied on Mitsubishi Heavy Industries for advice about whether a federal license amendment was required.

“Had MHI told SCE that a design change was needed to make the RSG’s safe, SCE would have approved it, even if that change would have required a license amendment,” Brown wrote. “MHI repeatedly told SCE that the design MHI proposed was safe, and the design did not require a license amendment.”

Mitsubishi did not respond to a request for comment for this story. The company in the past has said it could not have anticipated the unprecedented type of tube vibrations that occurred in exceptionally large generators commissioned by Edison.

The NRC has since confirmed that no license amendment was required for the San Onofre steam generators, and in fact, the issue may be irrelevant to the plant failure.

“The San Onofre steam generator tube degradation occurred as a result of issues introduced during the design phase that were unrecognized and, thus, were not considered in the licensee’s 10 CFR 50.59 evaluation,” the agency said in a March 2015 “lessons learned” memo about San Onofre.

Whether the amendment was required or not, investigators have probed whether the project would have received approval after undergoing such a review.

Elmo Collins, the former federal administrator who oversaw San Onofre until March 2013, told the NRC’s inspector general for an October 2014 report that if the license amendment review had been conducted, it is unlikely the steam generators would have been approved.

“The steam generators as designed were basically unlicensable,” he said. “We wouldn’t approve them.”

He said inspectors conducting a review would have noticed, in particular, the high “void fraction” of 95 percent when no other plant in the industry was above 90 percent.

“Some reviewer would have said this as an outlier,” Collins told investigators, “and we need to understand that.”

Disastrous outcome

Some of the first documents made public that raised questions about Edison’s oversight of the San Onofre project were written by Dwight Nunn, a now-retired company vice president. They surfaced in 2013, right before Edison decided to close the plant for good.

More than 10 years ago, Nunn wrote to the Mitsubishi Heavy Industries general manager.

“I am concerned that there is the potential that design flaws could be inadvertently introduced into the steam generator design that will lead to unacceptable consequences (e.g. tube wear and eventually tube plugging),” Nunn wrote. “This would be a disastrous outcome for both of us and a result each of our companies desire to avoid.”

Ray Lutz of Citizens Oversight, a San Diego nonprofit group fighting to reverse the San Onofre settlement, said the letter shows Edison could have prevented the failure.

“The Nunn letter really showed us that the utility knew they had a big problem,” Lutz said. “It’s clear when you read it that SCE was taking a very close look at everything going on with the design, so for them to say they didn’t know it could fail just isn’t true.”

For Edison’s part, it says Nunn’s letter shows how thorough the company was in making its concerns known to Mitsubishi, and gaining assurances from the manufacturer that the steam generators would be safe.

“Mr. Nunn’s questions could not have reflected a recognition that the design was flawed as the design did not even exist at that point,” Brown wrote. “On the contrary, his letter reflects the questioning attitude that is the hallmark of the strong safety culture in the nuclear industry. In response to Mr. Nunn’s questions, MHI assured SCE that it would carefully address the design of anti-vibration bars, recognizing the need to modify and improve the design from those it had developed for smaller RSGs.”

Experience invalid

While Edison and Mitsubishi executives traded correspondence in the early days of the project, engineers from the two companies convened in Japan to tackle the more technical issues. Notes from some of those meetings have been posted on the Edison website as part of the company’s effort to keep ratepayers informed about the shutdown.

The non-proprietary version of records from five days of meetings in March and April 2005 show how complicated the design challenges were and how detailed the discussions could get.

According to the public meeting notes, Edison was aware that Mitsubishi was venturing into new ground when it won the $680 million bid to design and manufacture the San Onofre replacement steam generators.

“MHI has experience with small (steam generators) and the SONGS RSGs have large U-bends, therefore the prior MHl experience is invalid,” notes from the first day of those meetings state.

On another page, one attendee makes this notation: “The SONGS OSGs (original steam generators) had tube wear problems so they don’t see why the RSGs will be any different. They suggest a comparative analysis of the OSG and RSG.”

Brown said Mitsubishi was responsible for designing the replacement steam generators, and that Edison challenged the process along the way. SCE pointed out, for example, that Mitsubishi was experienced building smaller steam generators and “MHI should not assume that a scaled up design would work,” she said.

“MHI agreed with SCE’s comment and repeatedly assured SCE that it was engaged in a rigorous evaluation of the safety of its design for the SONGS RSGs,” she wrote. “Unfortunately, unbeknownst to SCE at the time, MHI did not in fact live up to its promises, due largely to flaws deeply embedded in its proprietary computer codes.”

Edison’s presentation

In the run-up to a key meeting with federal regulators in 2006, Edison produced a 22-page slide show outlining its replacement steam generator project.

The presentation shows Edison planned to design, build, install and operate the RSGs at San Onofre before the close of 2010. The PowerPoint notes that Edison planned to replace the steam generators without going through the lengthy license-amendment process.

At the meeting, Edison touted the plant history and performance. The report does not include any reference to “void fraction” or other design challenges discussed by Nunn and other Edison officials prior to the presentation.

Hirsch, the UC Santa Cruz nuclear policy expert, said Edison “absolutely” should have reported its design concerns to federal regulators.

“It took NRC one day — one day! — to discover the computer error that was at the heart of the steam generator failure,” Hirsch said. “But because Edison tried to avoid a license amendment that would have required NRC review and a potential public license amendment hearing, and didn’t disclose to NRC problems like the void coefficient concern, NRC only did that review after the steam generator failed.”

Brown said there was no relevant concern to disclose to the NRC at the 2006 meeting, and the utility kept regulators in the loop throughout the project.

“We routinely shared status on the project with the NRC,” she said. “We addressed questions that they raised, and when they did inspections, we participated fully and provided them with information to address those questions. During the design phase, we also briefed the NRC headquarters staff at a high level.”

The utility told regulators at the 2006 presentation that the project had an “improved AVB design” and “improved materials for tube supports.” It would also use a stronger metal — thermally treated Alloy 690 TT — to construct the tubing.

The company said the new steam generators would hold 9,727 tubes each, a 4 percent increase, without adding to the height or diameter of the equipment. They would jump in weight from 620 tons to 644, Edison reported.

The new RSGs would be shipped by heavy-lift cargo from Kobe, Japan, to Long Beach, Calif., where an ocean barge would deliver them to Camp Pendleton.

Eleven months after it was equipped with replacement steam generators, Unit 3 was shut off due to the radiation leak. Its sister unit, offline that day for pre-scheduled maintenance, later was found riddled with the same excessive wear.

Edison and Mitsubishi are litigating their dispute through a private arbitration process. Edison is contesting a contract provision that caps at $138 million the liability for Mitsubishi.

“The RSG project was expected to extend the life of SONGS by decades,” Brown said. “Had the RSGs worked as MHI promised, utility customers would have benefitted.”