Innovations in modern science and computer technology are strongly linked together, and astronomy has unquestionably benefited from the advances in high-powered computing systems. However, the proposed Square Kilometer Array (SKA) of radio telescopes promises to push well beyond the current computing ability of the entire planet: if it works as planned, SKA will produce over one exabyte (260 bytes, or approximately 1 billion gigabytes) every day. This is roughly twice the global daily traffic of the entire Internet, and will require storage capacity at least 10 times that needed for the Large Hadron Collider.

The Netherlands Institute for Radio Astronomy (ASTRON) and IBM announced a collaboration today to develop the exabyte technology needed to support SKA. The initial investment of €32.9 million over five years will be used to investigate the types of new processors, power supplies, storage systems, and networking technology necessary to handle the amount of data needed. The DOME collaboration (named for the mountain in Switzerland, where the European IBM research center is located) will strive to develop fast, energy-efficient computers, which should benefit all computer users in the future.

As the name suggests, SKA will be a connected set of millions of radio telescopes with a collecting area of approximately one square kilometer. By comparison, the largest single telescope in the world, the Arecibo Observatory in Puerto Rico, has a collecting area of 0.073 square kilometers, while the Very Large Array (VLA) has a collecting area of about 0.013 square kilometers.

To achieve good resolution in astronomy, it's necessary either to build very large telescopes or coordinate multiple instruments into an interferometer, where signals from individual antennas are combined to achieve higher performance than any of the individual components can provide. (Radio light also has much longer wavelengths than optical light, so radio telescopes must be correspondingly larger to achieve the same sensitivity.)

The scientific goals for SKA include observations of the environment in which the first stars formed, approximately 400 million years after the Big Bang. To perform at the necessary sensitivity and resolution, SKA will produce a few exabytes of data every day of full operation. This will require not only short-term storage, but also processing and correlation of data from across the entire array of millions of telescopes. In other words, the challenge is not merely to collect and store the data, but to process at least some of it in real time.

The annual storage needs for SKA will be between 300 and 1500 petabytes (1 petabyte is approximately 1 million gigabytes), compared to the 15 petabytes per year of storage used for the LHC. However, while much of the data from the LHC is obtained in relatively short bursts of activity, SKA's collection will take place over more extended observation runs, requiring an entirely different set of methods to process it.

While handling data on this scale is possible with current technology simply by brute force, the energy requirement is prohibitive. Thus, the DOME collaboration will investigate new processors, optical networking techniques, and fast storage methods that are energy efficient. Both stacking chips in three dimensions to reduce the distance between processors and phase-change memory (which is faster than flash and requires no energy to maintain) are part of the plan, according to Chris Sciacca at IBM Research in Zurich. The testbed for the technology will be the LOw Frequency ARray (LOFAR), a collection of thousands of small radio telescopes in the Netherlands also managed by ASTRON.

The DOME project is obviously just beginning, so it is premature to state specifically what it will achieve. However, the advances in photonic communications and new acceleration technologies are promising, and this investment in "green supercomputing" is likely to benefit everyone in the long run. A lot of recent progress in computing technology has focused on greater energy efficiency, and the DOME collaboration should benefit far more than astronomy.

The Square Kilometer Array is a scientific collaboration between more than 20 nations. The projected date for the start of full operations is 2024.