After Q, the lab's engineers truly understood that neutrons are not neutral parties, so now they try to preempt problems. Before Los Alamos installs new equipment, like its Trinity machine, engineers perform a kind of cosmic stress-test, placing the electronics in a beam of neutrons—many more than cascade from the sky at any given time—and watching what happens. “We take parts and make them radioactive and make them crash,” explains Blanchard. They will also soon place neutron detectors inside the supercomputing center, to measure the strength of the storm. If you know how many neutrons you’re getting, and you know how they make computer parts behave, "you can predict the lifetime of your electronics,” says Suzanne Nowicki, a physicist in the lab’s space science and applications group.

Supercomputers are usually smart enough to know if something has gone wrong, to feel that flipped bit like you’d feel someone tugging on a single strand of hair. And when that happens, the system simply usually reports the error and rights itself. But sometimes, says Blanchard, the computer is more pessimistic. “I have an error. Too many bits flipped,” he mimics. “I can’t fix it, but I wanted you to know it happened.”

When that occurs at Los Alamos, they crash the computers—intentionally. It's like falling down on purpose when you're skiing, because that will hurt less than whatever else is about to happen. But you don't have to walk back to the top of the slope and start all over again: The engineers have created "checkpoints" throughout the quest to answers. It's like the save-spots in video games: If you die, you don't have to start all over. You start at the last spot you cached your achievements. Supercomputers can do the same kind of save.

The real problem, though, is “silent data corruption.” That's when the bits flip, and no one notices. The answer that you think is right may actually be a neutron-induced dream. That's why the preemptive work is so important: They know what to expect, how often, and can keep an eye out for it. At the same time, with that knowledge, the team hopes to turn what could have been silent errors into screaming errors. But if something does slip through, it's possible the flesh will catch it. Usually, Los Alamos doesn't say, "Here's your answer!" until an actual human checks out the results to see if they make sense.

That personal intervention happens partly because Los Alamos does crucial research on topics that affect many other persons. “The laboratory—the Department of Energy in general—studies climate change, new drugs, epidemiology, the spreading of diseases, wildfire modeling, all kinds of disease modeling, materials science, fragility of new metals,” explains Blanchard. And, as Blanchard adds after this list, the reason Los Alamos exists is because humans (some here, actually, at this lab) created nuclear weapons. “We’re a nuclear weapons lab,” Blanchard says. “Our job is stockpile stewardship. Our job is to make sure it’s secure and works as designed and doesn’t work when it isn’t supposed to.”

Because of nuclear test bans, the only legit way to stop worrying and learn to steward the bomb supply is to simulate—on a supercomputer—what’s going on inside. And so this place that concerns itself with radiation on Earth also must concern itself with radiation from space. Because whatever work supercomputers do in the future, one thing is certain: “They’re a bigger target every year,” says Blanchard.

More Great WIRED Stories