Operating some of the most complex machinery ever built can be a daunting task for the best of engineers. Especially when something breaks down hundreds of millions of kilometers away.

"Yes, my hands were shaking a little,” recalls Jean-Yves Bonnet. Engineer at the LATMOS, he is in charge of the gas chromatograph (SAM-GC), an instrument on board NASA’s Curiosity Rover used to analyze atmospheric and solid samples on Mars. On May 10th, following a long data crunching session, he and his colleagues from the GSFC uploaded new instructions to reconfigure the device, located a “mere” 350 million kilometers away.

Curiosity call home

Three months earlier, one of SAM’s six chromatographic columns—each dedicated to a different family of chemical components—had broken down. “We found an anomaly in column 5, the most used with 27 analyses already completed. The helium that carries the samples inside the column wasn’t passing through normally. We actually had a rock sample in there and lost part of the data,” explains Bonnet. All the researchers had to find the root cause were pressure and temperature measurements, as well as readings from two detectors inside the device.

You have to make sure the code doesn’t generate more problems. This is why the process took three months: first, to identify the problem, then write the repair code that would re-establish the carrier gas flow in the chamber, and finally test it on the replica of the instrument located at the GSFC in Washington, before sending it to Curiosity. “Since you only get one shot at this, you have to make sure the code doesn’t generate more problems, or renders the instrument useless.”

At the same time and still on Curiosity, ChemCam, a set of sensing instruments capable of taking both high-resolution images of soil samples (RMI) and determining elemental composition using a laser beam (LIBS) also experienced a malfunction. The component used to focus the laser had stopped working. “The ChemCam team’s solution was somewhat more complex, as it involved rewriting the operating system of the instrument entirely in order to use the high-resolution camera to determine the targets of the laser module,” adds Bonnet. On May 11th, data came back from Curiosity indicating that all systems were fully operational, much to the relief of teams back on Earth.





Known unknowns

“Such ‘remote’ fixes become even more complex when you add extraordinary time constraints,” explains UPMC professor and comet specialist Anny-Chantal Levasseur-Regourd, also from the LATMOS. She speaks from experience, as a co-investigator of several instruments on the Rosetta mission and on its lander Philae, whose landing on comet 67P a year ago captured the world’s attention. Concerning the Rosetta mission, the only glitch had to do with a thruster and harpoon malfunction on Philae, preventing it from securely anchoring itself to the planned landing site. “Instead of one landing, we got three for the price of one. This was of course one of the scenarios that had been explored. The worst-case had the lander completely bouncing off the comet, lost in space,” she explains.

Such ‘remote’ fixes become even more complex when you add extraordinary time constraints. Many instruments were operational during those rebounds, gathering data from three different areas. “Once we figured out that Philae had ended up in a dark and cold place, we needed to act very fast to ensure that all this data would be uplinked to Rosetta before its batteries ran out—about 60 hours,” she continues. “With very little information available, the engineers managed to make a last-ditch effort to change Philae’s orientation—known as attitude—to get this data transferred. This was definitely not business as usual.”

Little maintenance ahead

The main reasons that make communication in space so complex are no secret: poor data rates (the equivalent of a 56K modem for NASA’s New Horizons), huge distances (time delays for reception), and complex alignment (Philae and Rosetta, for example). This is why engineers continue to make these space probes and spacecraft more autonomous and resourceful, often with built-in contingency plans. Some researchers, like the ISIR team in Paris, are exploring surprising avenues. “We’re working on a code that allows machines to autonomously discover alternative behaviors when they partially break down,” says Antoine Cully, lead researcher on the project.

His team has created a system that allows a robot to test its physical capabilities and overcome various malfunctions. In this case, the researchers used a 6-legged robotic insect, whose objective was to keep on moving forward. “It’s a two-step process. The first consists in running a simulation sifting through all the possible ways it can articulate its legs—1047 possibilities in all—using an evolutionary algorithm selector. Following this virtual test period, the algorithm was by itself able to find 13,000 efficient methods of walking,” explains Cully. Researchers then proceeded to “damage” the device by taking off one, then two limbs, replacing a “foot” by a piece of wood, etc. “We made 5 different permutations, and each time, after loading up this code, the robot took less than 2 minutes to find a way to keep moving,” Cully says.

The most vivid example occurred within the simulation when all the “feet” were removed. “The machine flipped itself over, and continued ‘crawling’ on its knees,” enthuses Cully, who is now working on adapting this technique to multi-purpose robots. Although currently focused on industrial applications, the scientists are aware that such “adaptive” robots could prove extremely useful for rescue missions in hostile environments, or anywhere they need to operate without human intervention. “There is no reason why we shouldn’t think that future space exploration rovers or spacecraft could be given similar self-recovery code to continue operating for longer periods, or deal with time-critical situations,” he adds.

Levasseur-Regourd is more reserved. “Space missions are not like earth-bound laboratory testing that you can repeat over and over again,” she concludes. “They represent a unique opportunity, which we can only have if we try to do something that is truly exceptional—at the very limit of technical feasibility. Of course there will be glitches, but as we have seen, the overwhelming majority of these incredible projects are successes.”