The author of this page will appreciate comments, corrections and imagery related to the subject. Please contact Anatoly Zak .

Above: The flight version of the Phobos-Grunt spacecraft minus its main solar panels is being lowered into a vacuum chamber at NITs RKP test facility in Peresvet, north of Moscow, for thermal, vacuum and electric tests around beginning of June 2011. Credit: NPO Lavochkin

Previous chapter: Phobos-Grunt development in 2010

Completing the spacecraft

A very first indication that the Russian space agency intended to go ahead with the controversial launch of the Phobos-Grunt spacecraft in 2011 came in March, when the agency solicited bids to be submitted by May 20 for a contract to deliver a Zenit-2SB rocket for the mission from the manufacturer plant in Dnepropetrovsk, Ukraine, to the launch site later that year. Another published Roskosmos tender covered launch operations in Baikonur beginning in August 2011.

During his visit to NPO Lavochkin, Roskosmos head, Anatoly Perminov, reportedly witnessed a test of the fully assembled soil-collection mechanism.

In the meantime, unofficial reports from NPO Lavochkin still indicated that the project's Achilles' heel -- the flight control system -- had been far from completion. Six months before the projected launch, integrated electric tests of the vehicle were yet to be conducted and the simulated flight sequence needed for tests at the NITs RKP test facility had not been completed. When the tests of the flight sequence finally started, the flight control system reportedly was not able to score a single clean run of its entire flight program without experiencing some sort of problems.

As a time-cutting measure, the spacecraft developers reportedly discussed the possibility of launching the spacecraft with only partially completed flight software. The rest of the code responsible for later phases of the mission could be uploaded into the spacecraft's memory en-route. Obviously, this risky tactic was opened to criticism, however other sources said that all software had been ready for launch.

NPO Lavochkin was also finalizing the process of landing on Phobos -- one of the riskiest phases of the mission -- with the help from Applied Mathematics Institute of the Russian Academy of Sciences, IPM RAN, a veteran organization supporting the Russian space program. Algorithms of the landing were tested at a dedicated test stand at IPM.

To save cost, the Phobos-Grunt project had no dedicated prototype of the spacecraft for electric and radio tests, unofficial sources said. As a result, all testing of electric systems had to be conducted on the so-called "complex stand." However the facility featured only a limited number of onboard systems, with most components represented by mockups and prototypes. According to critics, it was not nearly enough for adequate development of a brand-new spacecraft. Moreover, as most components were still undergoing considerable upgrades, their prototypes on the test stand often did not reflect their actual performance. As a result, the final phases of testing and upgrades had to be conducted on the actual components of the flight vehicle, thus delaying the scheduled start of integrated tests of the fully assembled flight vehicle.

Moreover, blueprints for some components of the flight hardware started coming to the production arm of NPO Lavochkin only in December 2010. Even this documentation would have to be changed in parallel with hectic production efforts. As a result, the manfacturing and welding of some elements dragged until summer. Especially, many problems were reported with Kaluga's branch of NPO Lavochkin, which was responsible for solar panels of the cruise stage, PM.

Vacuum testing

As of 2010, the flight version of the Phobos-Grunt spacecraft was scheduled to undergo its most rigorous and critical tests in the VK600/300 thermal and vacuum chamber at NITs RKP test center in Peresvet in February 2011. This work could continue for up to three months. However in an interview with the Interfax news agency published on March 27, the Director General of NPO Lavochkin, Viktor Khartov, said that Phobos-Grunt was yet to go to Peresvet, following the completion of similar trials with the Spektr-R radio telescope satellite. During a meeting of the Chief Designer Council on March 30, 2011, it was reported that preparation for the shipment of the spacecraft to the thermal and vacuum testing facility was in the process of completion. Still, the official press-release issued after the meeting claimed that the work had been conducted on schedule. In an interview with Gazeta.ru published on April 13, the director of the IKI space research institute said that the entire spacecraft would be tested in the large vacuum chamber on April 15.

In reality, the flight version of the spacecraft was finally shipped to Peresvet by May 29, unofficial sources said. Only on June 2, did NPO Lavochkin confirm that the preparation for the complex electric tests in the thermal vacuum chamber at Peresvet had being conducted by the organization's team since the middle of May. At the time, engineers were completing the connection of ground cable lines, the installation of the thermal protection blankets, and of heat and vacuum sensors. The integration of all the spacecraft's major components and its installation into the vacuum chamber was planned for the near future, the NPO Lavochkin statement said.

On June 6, the head of NPO Lavochkin, Viktor Khartov, personally supervised the testing of the spacecraft in Peresvet, as it was placed into the VK600/300 chamber. Official photos published at the time showed Phobos-Grunt inside the vacuum chamber. Tests continued until June 20 and a day later, NPO Lavochkin announced that the spacecraft was removed from the vacuum chamber and was being prepared for a trip back to the design bureau. At the time, the shipment of the Phobos-Grunt to the launch site was promised at the end of the summer - beginning of fall. On June 27, the spacecraft was reported to be back at NPO Lavochkin. The company said that more electrical and vibration tests were planned in the near future.

Finishing payloads

Although most scientific payloads for the Phobos-Grunt mission were prepared for launch in 2010, several instruments had to be completed during 2011. Due to funding delays and production problems, the Tarusa branch of the Space Research Institute, IKI, had not been able to deliver the MANAGA instrument during 2010. Also, the MAL-1F mass-spectrometer developed at Vernadsky GEOKhI institute was yet to be delivered to IKI. (474)

Launch site preparations

On Sept. 29, 2011, the MDU propulsion unit for the Phobos-Grunt spacecraft arrived to the Yubileiny airfield in Baikonur. Following Kazakh custom procedures, the stage was to be moved into the processing building No. 40 at Site 31 for pre-flight operations, Roskosmos said. On Oct. 7, 2011, in the clean room at Site 31, technicians checked the pressure inside the tanks of the MDU unit and tested resistance in the wiring of its engine.

On Oct. 13, Roskosmos announced that the MDU propulsion unit was undergoing pneumatic tests in the clean room at Site 31. In meantime, at the nearby fueling station, technicians prepared the hardware for loading propellants and pressurized gases onboard MDU. In parallel, personnel of the TsENKI processing and operations center and KB Yuzhnoe design bureau started the preparations of the Zenit-2SB launch vehicle for launch at Building 41 at Site 42. In the first step, technicians removed transportation hardware from both stages of the rocket. On October 14, technicians started making electric and mechanical connections between stages of the launch vehicle.

Problems with flight control system

During the Spektr-R launch campaign in the middle of July 2011, the head of NPO Lavochkin, Viktor Khartov, told reporters that the Phobos-Grunt spacecraft would be delivered to Baikonur between Sept. 23 and Sept. 25, 2011, with its launch scheduled for Nov. 3-5, 2011. By mid-August this schedule reportedly slipped by around two days. However, as it transpired after the launch, the delivery of the spacecraft to Baikonur was delayed by a dual failure of a yet-to-be identified onboard system. It required a hectic overtime effort by NPO Lavochkin's production arm to complete an urgent replacement of the component.

In 2012, a report surfaced that the completion of a special test stand for checking a fully assembled flight control system, BKU, had been officially signed off only in August 2011. As a result, all previous checks had to be rushed onto the flight vehicle, requiring much more time and increasing the probability of errors. Even these inadequate tests clearly showed that BKU had not been ready for the mission. Continuous errors in software required frequent interruptions in testing and constant re-writes of the code. It was practically impossible to re-test the latest software patches. The testing had to be continued in Baikonur. (544)

In February 2012, it transpired that one of two processors in a pair of TsVM-22 computers in the probe's flight control system had failed and had to be deactivated during preparations for launch. To make matters worse, the second remaining channel in this computer started conflicting with its counterparts in the second remaining TsVM-22 machine. As a result, the decision was made to shut off one TSVM-22 computer entirely, even though it remained physically onboard. Therefore, Phobos-Grunt would have to fly with only a single operational computer, leaving no redundancy in the flight control system.

According to industry sources, none of rank-and-file engineers at TsNIIMash institute, responsible for certifying Russian spacecraft for flight, agreed to clear the Phobos-Grunt mission for launch. However top officials at the organization, whose signatures could really make a difference, were reportedly under heavy pressure from the agency leadership to give their green light to the ill-fated launch.

Delivery to Baikonur

Finally, after many delays, a cargo plane departed Moscow's Sheremetievo airport on the morning of Oct. 17, 2011, with the Phobos-Grunt spacecraft onboard, NPO Lavochkin announced. Probably indicating a severe pressure on the schedule, the company still did not officially announce the launch date of the mission, saying only that it was scheduled to lift off in the first half of November.

On the same day, the An-124-100 Ruslan aircraft of the Polyot company with the Phobos-Grunt touched down at the Yubileiny airfield of the Baikonur Cosmodrome. After standard custom procedures with Kazakh authorities, the spacecraft was unloaded on track platform and transported to Site 31 for pre-launch processing, Roskosmos said. Again, the agency's press-release said only that the launch of the mission had been planned in November.

On October 21, the MDU propulsion unit of the Phobos-Grunt spacecraft was transferred to the assembly building after being loaded with propellants during a previous week. In its turn, the planetary probe itself, was moved to the fueling station at Site 31 for loading of propellant and gases onboard the cruise stage and the return vehicle, Roskosmos announced. The agency also confirmed that the Phobos-Grunt mission was scheduled to launch on Nov. 9, 2011, at 00:16 Moscow Time.

The fueling of the probe's cruise stage with pressurized gas and propellant components was completed sometimes before October 27 and the Chinese YH-1 micro-satellite was also fueled around the same time. The fueling of the Earth return vehicle was scheduled for October 27 and was confirmed as completed on November 1. In the meantime, the MDU propulsion unit was integrated with the transfer section, which served as an interface between the spacecraft and the second stage of the Zenit-2SB rocket.

Final tests uncover serious problems

On November 1, inside the MIK-31-40 assembly building, joint industrial teams had conducted tests of opening of the probe's solar panels, Roskosmos said, without much details. However only after the launch of the ill-fated mission, not unexpected stories started surfacing about numerous problems which had been uncovered during final tests in Baikonur.

As it transpired, engineers found that the polarity for the steering control of the main engine had been mistakenly switched over in the flight control software of the main BKU computer. Nobody knew how such a major problem could've been missed during supposedly thorough certification tests in Moscow. Initially, managers decided to limit corrective actions to re-soldering of respective cables. As a result, six cables had to be reconnected on a spacecraft fully loaded with toxic and explosive propellants. Similar attempts to do electric repairs on the fueled vehicle led to a massive loss of life in Baikonur in the aftermath of the infamous Nedelin disaster in 1960.

However, after these repairs, it had become clear that all control signals from multiple cables had been channeled to the same driving mechanism. This mistake could not be fixed by re-routing cables, instead requiring complex changes in the flight control software. Last updates to the programming software had to be added right at the launch site, leaving no time for a new round of tests. The incident was reported to the head of the Russian space agency, Vladimir Popovkin. (534)

Final assembly

The final assembly of the payload section for the Phobos-Grunt mission started at Site 31 on November 2, with the rotation of the probe into a horizontal position, followed by the installation of the payload fairing. The transfer of the completed payload section to Site 42 for the integration with the launch vehicle was planned for the end of the same day, Roskosmos said. On November 3, Roskosmos released a series of photos showing the installation of the payload fairing onto the spacecraft. In the meantime, at MIK 41 assembly building a Site 42, personnel was preparing the Zenit-2SB rocket for the integration with the payload section, the agency said. After the delivery of the payload section to Site 42, it would be placed onto a specialized platform for the integration with the launch vehicle.

A State Commission overseeing the launch of the Phobos-Grunt mission held a meeting on November 5, during which space officials gave a green light to the rollout of the launch vehicle with the spacecraft to the launch pad. A railway transporter with the Zenit-2SB rocket and the Phobos-Grunt probe was scheduled to leave the assembly building at Site 42 next morning at 08:00 Moscow Time. The rollout did take place as scheduled on November 6.

On November 8, personnel at Site 45 started final preparations for launch: charging the spacecraft's batteries and checking temperature and pressure in the payload section of the rocket. A crucial meeting of the State Commission overseeing the launch was scheduled for 18:00 Moscow Time (9 a.m. EST), which would consider results of all tests and give a green light to the fueling of the rocket. Some half an hour before 19:00 Moscow Time (10:00 a.m. EST), Roskosmos confirmed that the State Commission had given go-ahead to the loading of propellant components onboard the vehicle.

Next chapter: Launch of the Phobos-Grunt spacecraft

APPENDIX

A summary of pre-launch milestones in the Phobos-Grunt project, as of October 2010 (432):

Development of soil-sampling system 2010 January-December Electrical tests of the flight version of the vehicle 2010 October-November Delivery of the YH-1 spacecraft from China 2010 Dec. 20 Final assembly of the flight vehicle, electrical tests 2010 December-2011 January Electrical tests of flight vehicle in vacuum chamber 2011 February-March Final operations 2011 July-August Delivery to the launch site 2011 September Launch window 2011 Nov. 5-25*

*As of August 2011; previously the window was expected to open as early as Oct. 28.

Chronology of the Phobos-Grunt project in 2011:

May 29: The flight version of the spacecraft was finally shipped to NITs RKT in Peresvet.

June 27: The Phobos-Grunt spacecraft is reported back at NPO Lavochkin after testing at NITs RKT in Peresvet.

Sept. 29: the MDU propulsion unit for the Phobos-Grunt spacecraft arrived to the Yubileiny airfield in Baikonur.

Oct. 14: Technicians started making electric and mechanical connections between stages of the launch vehicle.

Oct. 21: The MDU propulsion unit of the Phobos-Grunt spacecraft was transferred to the assembly building after being loaded with propellants during the previous week. In its turn, the planetary probe itself, was moved to the fueling station for loading of propellant and gases onboard the cruise stage and the return vehicle.

Oct. 27: The fueling of the the probe's cruise stage and Chinese YH-1 spacecraft is completed, with fueling of the return stage planned for that day.

Nov. 1: The opening of the Phobos-Grunt's solar panels is tested at MIK-31-40 in Baikonur.

Nov. 2: The final assembly of the payload section, installation of the payload fairing and the transfer of the payload section from Site 31 to Site 42.

Nov. 5: State Commission overseeing the launch gives "go-ahead" to the rollout of the launch vehicle with the Phobos-Grunt spacecraft to the launch pad.

Nov. 6: The Zenit launch vehicle with the Phobos-Grunt spacecraft rolled out to the launch pad at Site 45 in Baikonur.

Page author: Anatoly Zak; Last update: April 6, 2013