Managing hardware and storage needs; building custom, in-house applications; making information accessible via the Web—such tasks are the mainstays of IT work, so mundane that they're generally not worth talking about.

But science gives these routine tasks a fascinating twist. The hardware purchases support a compute cluster on the Caltech campus, while storage questions deal with the flood of data from genome sequencing. The in-house software reconstructs the text of ancient manuscripts. And the Web app helps people around the world follow our solar system and the hardware we've sent out to explore it.

For the people we've talked to, "working in IT" means working with some of the best minds on the planet on some of the hardest problems anywhere.

Translating physics

Doug Ellison wanted to be a rocket scientist, but one year of coursework in electrical engineering convinced him it was not to be. After graduating with a degree in graphic design, Ellison found himself working on software for medical education, but he never lost his fascination with space exploration and gradually found ways to express it. When NASA started posting online all the images taken by the Mars rovers, Ellison joined a small community of enthusiasts who stitched the images into panoramas, turned multiple exposures into real-color images, and even put together time-lapse videos of the rovers driving across the red planet.

The folks at NASA's Jet Propulsion Laboratory (JPL) were impressed—so impressed, in fact, that they offered Ellison a job. He left his native England for Southern California to work as the visualization producer for a team that takes space ship data and images and converts them into a Web app that lets anyone take a guided tour of the Solar System.

Ellison's biggest project is something called Eyes on the Solar System. The system relies on decades of data on various spacecraft, all distilled down into a file format called SPICE. SPICE-formatted data can produce exact details about a spacecraft's position and orientation at precise time points in the past, which is combined with maps of the planets and moons of our Solar System, creating a complete 4D environment (the three dimensions of space, plus time). The team also updates the system as more data comes in; Ellison specifically mentioned MESSENGER's study of Mercury as requiring a major update.

How well do you have to know physics to make all of this work? "We have a grasp on what the physics is doing, but we're fortunate that people far cleverer than we are do the heavy lifting," Ellison joked. "I'm a translator, and one of my languages is science."

To serve the data over the Web, it's compressed a bit but remains "more than accurate enough," according to Ellison. To display the environment in a Web browser, the team uses the Unity3D game engine, which works on a wide variety of systems. "As long as you're not on old school decrepit integrated graphics, you'll be fine," he added.

The end result is a system that lets anyone anywhere on the planet recreate historical events, like Voyager's flyby of Saturn, or track the progress of missions like the next rover's journey to Mars, all with full-color reconstructions of the spacecraft and their destinations.

The system has been used when a JPL mission ends up on TV, and it's used by scientists in presentations. But it's used most often by people who simply access it over the Internet. "We're still trying to figure out who our users really are," Ellison told Ars, noting "our users are not [the JPL management]." He knows that the tool gets used by space enthusiasts and in classrooms, and that people tune in to watch major events like comet flybys, but there's a steady stream of traffic even on days where nothing much happens.

Eliison has his dream job, in which he gets to work with brilliant people—some of whom he can still manage to surprise. "We're in a huge, political, bureaucratic machine, but there's this little bit of excitement at the core of JPL," he said. "You do this little demo… and there will be this gasp, because they [the audience] didn't appreciate how things work, how the Solar System operates."

Storing the human—or platypus—genome

Ellison measures the data used for his work in gigabytes, but science often demands far greater capacity. Across the country at the Broad Institute of Harvard/MIT, Matthew Trunnell was already dealing with a storage system that weighed in at 200 terabytes back in 2006. The reason? The automated systems built to sequence the human genome kept pouring out data for other projects, like the effort to sequence 1,000 human genomes or to capture the genomes of cancer cells. 2006 was also the year that the Broad Institute got the first of its "second generation" sequencing platforms, which only made the storage situation worse. Now, the Broad's data set sits at 10 petabytes and growing.

"Our data volumes take us completely by surprise," Trunnell admitted.

Trunnell got his start simulating fluid dynamics in oceanography. He only switched to working on bioinformatics when he moved over to the private sector but immediately found it fascinating. Eventually, he moved to supporting the people who generate the data and perform the analysis because, he said, "I like to build things but hate to run things."

What he gets to build is enough capacity to store and analyze thousands of genomes. Along with that 10PB of storage, there's a Linux cluster of 2,500 cores just dedicated to the process of assembling genomes out of the data pouring in from hundreds of automated sequencers. Such assembly is a memory-intensive task, so some of the machines involved are outfitted with a half-terabyte of RAM. Once assembled, the data has to be analyzed—which takes another 6,000-core cluster.

"IT changes quickly, and laboratory technologies are changing very fast, and the intersection between those is moving faster than either of them," Trunnell said.

Most of the processing can be split into jobs of an arbitrary length, which makes clusters an effective solution. But over time, two trends are changing the way the Broad thinks about hardware. First, the increasing number of cores makes newer nodes relatively "fat" (as Trunnell put it), meaning they require more RAM and local storage to function well. Second, as the data gets larger, getting it all to the right machine has become a growing challenge.

All that computing power goes to support three types of users. Some are just doing what you might call "traditional" genome analyses, which have been around for a while now and have fairly predictable requirements. Some of the newer projects, like the Cancer Genome Atlas, are very much works-in-progress, though, with new rules. "The focus is research—it's still at scale, so they are making very large scale use of computing resources and data, but the needs are much more fluid," Trunnell said. The computational methods are still being developed, the data is still being explored, so support requires a mix of large scale computing and a willingness to deal with rapidly developing needs.

The last group of users are typically students and post-docs who do more limited projects but who still need access to big hardware. "The needs are frequently small, but the impact is sometimes large because we have a large computing resource that is available to everyone," Trunnell said. "Groups that aren't used to using computation at scale can sometimes get carried away."

For Trunnell, the biggest appeal of the job is that he gets to work with everyone at the Institute at one point or another, so he gets a really "broad" picture of the work that's going on. And some of that work is truly significant. "The insights we're gaining into cancer right now are likely to have a tremendous impact on cancer therapeutics down the road," he said. "But being able to see these develop now is very exciting."