Previous Proton mission: SES-6

The ill-fated Proton rocket lifts off on July 2, 2013, at 06:38:21.585 Moscow Time (July 1, 10:38 p.m. EDT). The rocket crashed approximately 32.682 seconds later, Roskosmos said on July 18, 2013.

Russia's Proton crashes with a trio of navigation satellites

Published: July 1; updated: July 2, 3, 4, 5, 9, 11, 15, 18, 19; 23; Aug. 11

Russia's Proton rocket crashed less than a minute after its liftoff from Baikonur, Kazakhstan. A Proton-M vehicle No. 53543 with a Block DM-03 (11S-86103) upper stage lifted off as scheduled from Pad No. 24 at Site 81 (launch complex 8P-882K) in Baikonur Cosmodrome on July 2, 2013, at 06:38:21.585 Moscow Time (on July 1, 10:38 p.m. EDT).

The rocket started veering off course right after leaving the pad, deviating from the vertical path in various directions and then plunged to the ground seconds later nose first. The payload section and the upper stage were sheered off the vehicle moments before it impacted the ground and exploded. The flight lasted no more than 30 seconds.

The Russian space agency's ground processing and launch contractor, TsENKI, was broadcasting the launch live and captured the entire process of the vehicle's disintegration and its crash. Half an hour after the accident, a report on Russian web forums said that a team in the launch control bunker near the launch pad had been in communication with the rest of the space center and apparently had been unharmed. The launch vehicle reportedly crashed near another launch complex for Proton rockets at Site 200.

Since the emergency cutoff of the first stage engines is blocked during the first 42 seconds of the flight to ensure that the rocket clears the launch complex, the vehicle continued flying with its propulsion system firing practically until the impact on the ground.

According to the local Interfax-Kazakhstan news agency, the launch vehicle crashed one kilometer from the launch complex. However veterans of the facility looking at the video of the accident, estimated the distance at around five kilometers. It was later established that the rocket fell slightly more than one kilometer southeast of Pad No. 24, short of the main rail line connecting the Proton launch pads with the rest of the center and not far from a Silo facility at Site 175.

Aftermath

Around an hour after the accident, a Russian Vesti 24 TV channel reported that Kazakh authorities had considered evacuating population from the area around the crash site due to a possible danger from toxic propellants onboard the rocket. The main residential area of Baikonur Cosmodrome is located 57 kilometers southeast of the Proton launch area.

During the evacuation, some of the personnel at Baikonur took photos of a reddish cloud spanning over the main road to the facility. In Baikonur itself, the city's administration advised residents not to leave their homes, deactivate air conditioners and close tightly all doors and windows due to "a cloud of unburned propellant moving toward towns of Baikonur, Akai and Tyuratam." Residents of Kazalinsk and Karmakshinsk districts were also notified about the accident. Many stores and other public facilities were reportedly closed.

Fortunately, starting rain apparently helped to dissipate the cloud, Kazakh media reported. Also, eyewitnesses indicated that the cloud was moving mostly east of the crash site and thus did not pass directly over the residential area.

The access to the site was blocked by an emergency team and firefighters, however there was fire at the site, Interfax reported.

In Baikonur, Site 2, located at the center of the launch facility and east from the crash site, was evacuated in the first few hours after the accident, however personnel was allowed back later in the day. All restrictions in the residential area were lifted before 5 p.m. Moscow Time on July 2.

Impact site

On July 3, a joint team of environmental experts from Russian TsENKI ground support center and Kazakh ecological service reviewed the impact site. Specialists determined that 4.9 hectares of grassland was affected by a fire following the accident. The impact of the rocket left a 40 by 25-meter crater with a depth of up to five meters. The coordinates of the site were determined as:

46 degrees 3 minutes 38 seconds North latitude;

62 degrees 59 minutes 43 seconds East longitude.

This coordinates did confirm that the rocket barely missed the main railway line connecting the Proton launch complex at Site 200 with the rocket's processing complex and fell not far from a triple UR-100NU missile silo facility at Site 175.

Environmental services were reported taking air and soil samples around the crash site, with three chemical defense posts deployed in the area. No significant contamination was found, the Interfax reported. Kazakh news web site Kaztrk.kz quoted the nation's chief medical safety specialist Zhandarbek Bekshin as saying that thanks to an absence of wind at the time of the crash, toxic gases did not spread very far. Moreover, most of the toxic propellant burned during the explosion and rain helped to neutralize remaining chemicals. Bekshin confirmed that all toxicity tests in near-by towns of Tyuratam, Akai, Dzhusaly and Baikonur turned out negative.

According to the Interfax, the crash site remained surrounded by the police and emergency service workers as late as July 5. Daily air and soil sampling were continuing at the site and along the path of air flow, however concentrations of toxic propellant components did not exceed safe levels, the agency reported.

Press-service of the Kazakh space agency then announced that on July 8 special chemicals for the detoxication of the impact site had been delivered from Almaty, Shymkent and Astana. Hydrogen peroxide and iron complexonate were prepared, while the Russian side formed a team for the operation, prepared the plan of the cleanup work and deployed a personnel cleanup facility at the site during the soil detoxication effort conducted during July 9 and July 10, Kazkosmos said. A total area of 13,100 squre meters was treated with the detoxication solution. Follow up soil tests were planned after 10 days.

Investigation

Soon after 9 a.m. on July 2, Interfax posted a report quoting telemetry from the mission, according to which first deviation of the rocket along the pitch axis started four seconds after the liftoff. Oscillations increased at T+12 seconds and the emergency engine cutoff command was issued after 17 seconds in flight. The vehicle then started disintegrating and crashed 1.5-2 kilometers from the launch command post 32 seconds after the liftoff. (Editor's note: The claim about the engine cutoff is likely an error, as it is clear from the video that the propulsion system continued operating.) However within an hour, GKNPTs Khrunichev, the Proton developer, repeated the same claim in its press-release on the accident.

Around 8 a.m. in the morning on July 2, Roskosmos posted a statement on its web site about the formation of an investigative commission led by Deputy Head of the agency Aleksandr Lopatin. Almost simultaneously, Kazakh officials announced that the accident was an internal Russian affair, since the rocket had crashed within the territory rented by the Russian Federation. However later during the day, it was reported that both Kazakh and Russian governments formed their commissions in addition to an investigative team formed by Roskosmos.

On July 2, Proton's commercial operator, the International Launch Services, ILS, announced a formation of its own Failure Review Oversight Board, FROB. The FROB will review the commission’s findings and corrective action plan, in accordance with U.S. and Russian government export control regulations, ILS announced. According to the company, the FROB will include representatives from ILS customers, insurance underwriters and technical experts from the industry.

The ILS also reported that the impact of the failed rocket "occurred in a safe area that was evacuated for the launch and all personnel are reported to be unharmed. From early reports, there was no damage to either launch Pad 39 or 24, near the impact area; there is only minor damage to nearby buildings," ILS said.

Reconstructing a sequence of events

By July 4, engineers in Baikonur deciphered available telemetry from the failed rocket. Early in the process, one source reported an emergency cutoff in one of six engines on Proton's first stage in the first few seconds of the failed launch. Other unofficial sources then elaborated that a failed steering mechanism, known by Russian abbreviation as RM, placed the engine into an extreme position making it too difficult for the flight control system to correct a wrong direction of thrust with remaining five engines.

By the end of July 2, it became known that the liftoff of the ill-fated rocket had taken place 0.4 seconds ahead of schedule, potentially forcing the vehicle to start its flight with its engines at less than full thrust. According to the flight program, the rocket had to lift off at exactly 06:38:22.000 Moscow Time, with an acceptable deviation of 0.05 seconds earlier or later. However, the flight control system got an apparently faulty signal that the actual liftoff had taken place at 06:38:21.585, or 0.415 seconds too soon.

As a result, the rocket's flight control system could activate an emergency flight sequence designed to take the vehicle away from the launch complex. The telemetry also showed the temperature near the engine at 1,200 degrees, or three times higher than normal, likely an evidence of the fire in the area. Finally, the emergency engine cutoff signal for at least one engine was issued just four seconds into the flight, not at T+17 seconds as was previously reported.

On July 4, a source at GKNPTs Khrunichev reported on the online forum of the Novosti Kosmonavtiki magazine that an interface plate connecting a series of cables from ground equipment to the aft end of the launch vehicle, had separated earlier than planned. The plate, designed to shift by around 5 millimeters, normally trails the rocket for few millimeters and separates as the vehicle rises above the pad. However in this case, it apparently moved by as much as 11 millimeters before the rocket had a chance to leave the pad. As a result, all electrical connections between the pad and the rocket were severed, while the vehicle's engines were yet to develop their full thrust. At that point, the engines could still propel the rocket into the air, but could not keep it in stable flight. (According to the telemetry, the pressure inside the combustion chambers of the engines was 90 kilograms per square centimeter, instead of required 150 kilograms per square centimeter.) The flight control system could interpret such a situation as an emergency, (even if the rocket was still standing on the launch pad), and sharply throttle all engines to a maximum thrust in order to prevent the vehicle from falling onto the launch pad. In turn, the sharp increase in thrust could cause a fire detected by temperature sensors.

If confirmed, such a scenario would make the cable interface plate of the launch pad a culprit in the abnormal liftoff and the subsequent crash. The exact reason for the plate to go down was not immediately clear, but it could be due to its wrong installation or a mechanical failure. The erratic behavior of the rocket during its short flight also remained to be explained under such a scenario. According to one theory surfaced on July 6, the flight control system was receiving reversed readings of angular velocity from onboard sensors due to their wrong wiring, however the condition of the debris after the crash would not allow to confirm such a possibility.

On July 4, the head of the Russian space agency, Vladimir Popovkin did confirm in an interview with the official ITAR-TASS news agency that a failure of launch pad hardware was now considered among possible culprits along with the propulsion system and the flight control system of the Proton rocket. Popovkin estimated that the investigative commission would find the cause of the accident by the end of July.

Culprit found

By July 9, it is transpired that investigators sifting through the wreckage of the doomed rocket had found critical angular velocity sensors, DUS, installed upside down. Each of those sensors had an arrow that was suppose to point toward the top of the vehicle, however multiple sensors on the failed rocket were pointing downward instead. As a result, the flight control system was receiving wrong information about the position of the rocket and tried to "correct" it, causing the vehicle to swing wildly and, ultimately, crash. The paper trail led to a young technician responsible for the wrong assembly of the hardware, but also raised serious issues of quality control at the Proton's manufacturing plant, at the rocket's testing facility and at the assembly building in Baikonur. It appeared that no visual control of the faulty installation had been conducted, while electrical checks could not detect the problem since all circuits had been working correctly.

On the Proton rocket multiple DUS sensors are clustered into modules called Blocks of Damper Gyroscopes or BDG in Russian. They are designed to provide navigational information to Proton's sophisticated multi-channel flight control system. A total of three BDG modules are mounted on a special platform on the body of the rocket. The evaluation of the recovered platform showed that three BDGs responsible for the pitch movement of the rocket were installed correctly, however all three BDGs responsible for the course (yaw) axis were rotated 180 degrees. Since all three BDGs had been installed incorrectly, the three-channel flight control system could not filter out faulty data by comparing it to correct information coming from the majority of sensors.

The improper installation apparently required some considerable physical effort, which, somehow did not raise any alarm at GKNPTs Khrunichev's assembly plant in Moscow. Investigators immediately looked at already assembled Protons, including those in Baikonur, but did not find such an anomaly.

The BDG modules are developed at the Zvezda enterprise on the Gorodomlya Island and manufactured by PO Korpus in the city of Saratov. Both organizations are divisions of the NPTs AP design bureau, a prime contractor responsible for the flight control system of the Proton rocket and its Block DM-03 upper stage.

By July 13, investigators simulated the improper installation of the DUS angular velocity sensors on the actual hardware. As it turned out, it would be very difficult to do but not impossible. To achieve that personnel would need to use procedures and instruments not certified either by the design documentation or the installation instructions. As a result, the plate holding the sensors sustained damage. Yet, when the hardware recovered from the accident was delivered to GKNPTs Khrunichev, it was discovered that the nature of the damage to the plate had almost exactly matched the simulated version.

The improperly installed sensors apparently caused the rocket to sway from site to side during its vertical liftoff and left the vehicle completely uncontrolled as it attempted to steer itself along the preprogrammed path. According to the flight sequence, the rocket had to roll slightly and pitch in the direction of the flight. The roll maneuver apparently started, however, grossly distorted pitch commands caused steering mechanisms on the engines swing them into extreme positions from one side to another causing chaotic movement of the rocket beginning from the 12th second of the flight. By T+18 seconds, the steering mechanisms could no longer keep up with commands to swing engines from one side to another several times every second.

Roskosmos confirms key facts about the accident

On July 18, Roskosmos confirmed all key facts about the accident reported here previously. The agency's statement stressed that the launch vehicle did lift off 0.4 seconds earlier than scheduled, however the following analysis and simulations of the flight had shown that this situation could not had caused the accident. At the same time, the agency confirmed that the launch vehicle had lost stabilization along its course (yaw) axis, as a result of the improper installation of the DUS angular velocity sensors 180 degrees away from their correct direction. The investigation revealed that three out of six DUS instruments had shown marks of force applied to their docking surfaces and post-accident simulations conducted on mockups of the system had left similar impressions, Roskosmos said.

The agency concluded that existing methods of quality control during ground preparations and testing (of the rocket) according to current design, technical and operational documentation had not allowed to detect the improper installation of DUS sensors and, therefore, the defect had had a manufacturing nature.

The investigation commission had still continued its work to determine the cause of the premature separation of electric interfaces of the launch pad (from the rocket), Roskosmos announced.

Flawed assembly process

The official statement by Roskosmos on July 18 was preceded by a press-conference of the investigation commission chairman Aleksandr Lopatin. In addition to reiterating the known facts, he reported that the assembly of the failed Proton had been completed at GKNPTs Khrunichev in December 2011 and the rocket had remained in storage until May 2013, before being delivered to Baikonur. The wrong installation of three DUS sensors responsible for the course (yaw) correction took place on Wednesday, Nov. 16, 2011. Lopatin explained that a pair of five-millimeter pins on the mounting platform for DUS sensors are designed to help the technician in the correct placement of instruments, however with a certain effort it is possible to mount the sensor without those pins fitting into their holes and still attach it securely with fasteners. Moreover, it was possible to insert all incoming color-coded cables in their correct sockets, despite a wrong position of DUS sensors.

Lopatin noted that the particular installation process is very laborious and requires considerable skill from the electrical specialist, who has to access the location via a pair of half-a-meter hatches in the rocket. Even though the operation is classified as "under special control," there was neither established procedure for video or photo documentation of the process or its inspection by an outside agency. However the technician's supervisor and a quality control specialist were supposed to check on the completion of the installation. All three people involved in this process did leave their signatures in the assembly log.

Lopatin stressed that along with a human error, the investigation commission identified deficiencies in the installation instructions and in the mechanical design of the hardware, which both contributed to the problem. For example, the mounting plate lacked an arrow which would match the direction of an arrow on the DUS unit.

Way forward

Lopatin said that investigators had been inspecting two flight-ready Proton vehicles in Baikonur, as well as a pair of fully assembled rockets at GKNPTs Khrunichev in Moscow to ensure that they were free of similar issues.

The commission also developed measures aimed to exclude the recurrence of similar problems in the future. In a broader sense, recommendations were made to the agency's research institutions to re-evaluate the entire technological process adopted within the industry for any other quality control issues. A new system of photo and video documentation, which was just recently introduced into the manufacturing of the Briz upper stage, would now be expanded to the entire rocket-assembly process, Lopatin promised.

Lopatin estimated that after all the corrective measures, Proton-M rocket could return to flight by September. However he admitted that it would be difficult to make up the original schedule of launches for 2013.

Implications

It was the first time in the post-Soviet history that the Proton rocket, a largest vehicle in the current Russian rocket fleet, crashed in the vicinity of its launch facility. On April 2, 1969, a Proton rocket carrying a Mars probe crashed shortly after lifting off from the same Pad 24.

Serious implications for the Russian space program and its rocket industry are practically inevitable, however there are no alternative to Proton until Angara becomes operational later in the decade at the earliest.

Political fallout

Shortly after the accident, Russian media reported that authorities started a criminal investigation in Baikonur under Chapter 1 of Article 216 of the Criminal Code entitled "Violations of safety rules during mining, construction and other works leading to a significant damage." Unofficial source at the site did confirm that criminal investigators had appeared at various facilities.

Also, Russian Vice Prime-Minister Dmitry Rogozin promised to make "hard conclusions" from the investigation, which among other things "would identify those who despite numerous government requests failed to deal with many issues of quality control," Rogozin said. According to Rogozin, the failed vehicle had been manufactured and delivered to Baikonur before his administration restored the military certification of rocket technology.

On July 8, Rogozin announced a formation of an expert engineering group, which would not include representatives of the Proton's contractors, but would be responsible for the review of the entire manufacturing process and quality control procedures during the production of the vehicle.

On July 7, Russian President Vladimir Putin personally discussed the issues around Baikonur with Kazakh President Nursultan Nazarbaev, during his visit to the former Soviet republic.

In the meantime, on July 12, Vice Prime Minister Dmitry Rogozin promised to deliver the Russian government the results of the investigation into the Proton crash by the beginning of August. Rogozin was quoted as saying that it was too early to talk about the causes of the crash and all information on the accident appearing in the press had been expert estimates at best but could had been an attempt to mislead the investigation at worst. Rogozin met with the Roskosmos leadership and stressed the unacceptability of dissemination of any unverified information due to huge social resonance of the accident.

Kremlin provides final details on Proton accident

On Aug. 5, 2013, Russian government released a transcript of an investigative commission meeting chaired by Dmitry Rogozin. The head of investigation, Aleksandr Lopatin, reported that a Proton rocket for its 47th GLONASS mission had been manufactured in accordance with a federal contract between Ministry of Defense and GKNPTs Khrunichev signed on March 20, 2010. During its production, a total of 19 permissions were issued for changes in the standard design and manufacturing documentation of the launch vehicle.

Lopatin reiterated that all pre-launch activities had been proceeding normally until around 0.4 seconds before liftoff, when the emergency flight algorithm was activated. Around 6.8 seconds after the liftoff signal, the telemetry showed a sharp increase in the movement of steering mechanisms in engines No. 1, 3, 4 and 6, with their actuators reaching maximum angles. At T+7.7 seconds in flight, gimbal angles along the yaw axis reached their maximum possible angle of 7.5 degrees. Practically from the beginning of the flight, an unstable process of deviation from the correct yaw axis was observed. At T+12.7 seconds in flight, a signal indicating the exceeding of the maximum allowable angle was issued, as the stabilization system (of the rocket) was no longer able to control the yaw. As a result, at T+12.733, the "launch vehicle failure" command had been generated, Lopatin reported.

He then reiterated that the telemetry analysis had confirmed that the anomaly in the movement of the vehicle along the yaw axis had been caused by an abnormal operation of angular velocity sensors, DUS, in the PV-301 instrument unit. A total of six PV-301 units are mounted in two groups on a platform in the aft section of the second stage of the Proton rocket. Three units are responsible for the pitch and three for yaw axis of the flight trajectory.

Lopatin also confirmed that two out of three incorrectly installed yaw DUS sensors had been identified after their recovery from the crash site, thanks to remnants of red and yellow paint. The third such instrument was not positively identified because its paint cover had been completely burned.

The commission recommended NPO Technomash design bureau to compile a list of operations in the rocket industry that would require photo and video documentation and add respective changes to the existing manufacturing standards, known as OST. GKNPTs Khrunichev and NPTs AP design bureau were assigned to modify the PV-301 instrument and its holding platform to exclude the possibility of its incorrect installation. The commission also required to introduce photo and video documentation of the installation process and introduce additional checks of cable connections leading to the instruments.

Putting things in perspective (Editor's note)

Even though leaders of the Russian space industry could certainly do some soul-searching in the wake of the latest launch failure, general public and some politicians should probably put things in perspective, before making loud pronouncements about punishing those "responsible." Practically all great space powers in the world go through periods of multiple failures from time to time. This is the nature of modern space technology. The Russian space program, still emerging from a devastating post-Soviet crisis, currently maintains an extremely high rate of launches, in fact, for the first six months of 2013, Russia launched more space vehicles than all other nations combined!

It also should be noted that nobody was hurt in the accident and no third-party property was damaged, proving that the launch was conducted safely under the circumstances. The failed Proton rocket carried serially produced replenishment satellites for an already operating constellation, not a unique space station module, or a planetary probe, which takes 20 years to build. Without undue political interference, Russian engineers could probably find the problem and put Proton back in business in two or three months.

Nominal flight program

During the ill-fated mission, Proton was carrying three GLONASS-M satellites in the 49th mission to deploy and maintain the satellite navigation network. In addition, it is also a second attempt to inaugurate a modified version of the Block D upper stage, designated DM-03, after the first vehicle of this type doomed the 43rd GLONASS launch in December 2010 due to a propellant overload caused by a human error during fueling.

Following the liftoff, the Proton-M rocket was to fly due East to reach an initial parking orbit with an inclination 64.8-degrees toward the Equator. After the separation from the third stage of Proton, the Block DM-03 upper stage should fire its main engine to insert itself and its three payloads into a highly elliptical orbit. For the next three hours the stack will climb to an apogee of this orbit, where Block-DM-03 would fire again, this time to make orbit circular. Around three minutes later, a trio of satellites would be released from the stage.

The loss of three satellites was estimated at around $200 million, the Interfax news agency reported, however there were conflicting reports on whether the mission was ensured.

This launch was previously scheduled to take place as early as November 2012, however it was delayed to 2013 sometimes after beginning of May 2012.

Next chapter: RD-253 / RD-275 engine

APPENDIX

A GLONASS mission timeline on July 2, 2013:

Event Moscow Summer Time EDT Resulting orbit Liftoff 06:38:22 10:38 p.m.* N/A Payload section separates from Stage III 06:48:06 10:48 p.m.* 179.2 by 184.4 kilometers First firing of Block DM-03 engine (397.6 seconds) 07:14:29 11:14 p.m. * 216.2 by 19,160.6 kilometers Second firing of Block DM-03 engine (165.4 seconds) 10:07:38 2:07 a.m. 19,137 kilometers (circular) Separation of satellites from Block DM-03 10:10:48 2:10 a.m. 19,137 kilometers (circular)

*July 1

Page author: Anatoly Zak; Last update: August 21, 2015