Spencer Klein is holding a thick glass ball the size of a watermelon and it is stuffed with electronics. For 10 minutes or so, he turns it over in his hands and talks through what it does, how it works and the brutal environment it can withstand. This last point turns out to be key. Over the past half-decade, more than 5,000 of these objects have been shipped to the south pole, strung together like beads, and buried deep in the Antarctic ice sheet.

Klein is a physicist at the Lawrence Berkeley National Laboratory that sits high on the hills overlooking the University of California's Berkeley campus and beyond to San Francisco Bay. The glass ball in his hands is a "digital optical module" (DOM), an exquisitely sensitive light detector that lies at the heart of what must be one of the most ambitious projects in the history of science. By freezing these modules into the ground around the US Amundsen-Scott south pole station, on the high plain of Antarctica, Klein and his colleagues have turned a cubic kilometre of pristine polar ice into an enormous cosmic observatory.

The $272m (£170m) IceCube instrument is not your typical telescope. Instead of collecting light from the stars, planets or other celestial objects, IceCube looks for ghostly particles called neutrinos that hurtle across space with high-energy cosmic rays. If all goes to plan, the observatory will reveal where these mysterious rays come from, and how they get to be so energetic. But that is just the start. Neutrino observatories such as IceCube will ultimately give astronomers fresh eyes with which to study the universe.

The frigid conditions at the south pole meant construction teams could only work on IceCube between November and February each year when ski-equipped planes can safely make the 1,800-mile round trip to the research station. The DOMs are designed to run on precious little power, a measly 5W each, but even so, it takes 10 planeloads of fuel to run IceCube for a single year.

The final digital optical module (DOM) is prepared for the telescope.

The final piece of the observatory was put in place a week before Christmas when engineers used a hot-water drill to melt the last of 86 holes in the ice. The holes reach a depth of 2.5km (1.5 miles) and down each is lowered a string of 60 DOMs that are locked in place when the water in the hole refreezes. The pressure is so great at these depths that air bubbles are squeezed out of the ice, leaving it almost perfectly transparent.

Physicists on the IceCube project are now completing a series of checks on the latest additions to their bizarre instrument to see if the equipment survived the ordeal of being installed. Assuming it has – and only a couple of DOMs have failed in the project's history – the instrument will soon swing into action and its search for cosmic rays will begin in earnest. "Our best calculations show that we need an instrument this size to have a good chance of seeing these cosmic ray sources," says Klein. "Now we're done, we have it."

An Austrian-born scientist called Victor Hess discovered cosmic rays 100 years ago. In a series of hot-air balloon flights, Hess measured the radiation around him at altitudes up to and beyond five kilometres. As he rose up through the atmosphere, radiation levels initially fell, but then rose steeply until they were double that at sea level. Hess reasoned that radiation must somehow reach Earth from outer space.

Cosmic rays are now known to be highly energetic particles that originate in outer space and bombard our planet from all directions. Most are made up of charged particles, such as metal ions, but these are of little use to space scientists hoping to discover the origins of high-energy cosmic rays. Charged particles are deflected by magnetic fields as they race across space, making it hard, or impossible, to retrace their route and locate their cosmic birthplace.

Neutrinos are different. Produced alongside cosmic rays in outer space, neutrinos are uncharged and pass through normal matter almost entirely unhindered. Instead of being pushed and pulled around as they head towards Earth, neutrinos move in a straight line, giving scientists a good chance of tracing them directly back to their origins.

The most energetic cosmic rays seen in nature pack far more punch than any particle accelerator has ever achieved on Earth. "Some carry the same amount of energy as a well-hit tennis ball," says Klein. "To put that in context, if you wanted to build an accelerator that energetic, with the same technology they use at the Large Hadron Collider [at Cern, in Switzerland], you would need a ring of magnets the size of Earth's orbit around the Sun."

Scientists have some ideas about what cosmic events might produce these extraordinarily energetic cosmic rays. They could be driven by shockwaves emanating from exploding stars, or be propelled from supermassive black holes that sit at the centres of galaxies, gobbling up stars and other objects in the vicinity. In one scenario, cosmic rays stream out when a black hole collides with a neutron star.

"The biggest puzzle about cosmic rays is that they are the highest-energy particles we can see in the universe and yet we don't know what makes them. We have ideas, but it remains one of the outstanding mysteries of physics. What we want to find out is, how is nature doing this?" says Subir Sarkar, an astroparticle physicist at Oxford University and leader of the British team that works on IceCube.

IceCube is not the first neutrino observatory to be built by scientists, but it is by far the largest. In 1987, three neutrino detectors, constructed in caverns in Japan, America and the Caucasus, became the first to spot a few handfuls of neutrinos that sprayed out of a supernova called 1987A, which exploded in the Large Magellanic Cloud, a neighbouring galaxy to ours. In Siberia, a Russian-German team has lowered cables carrying 192 light sensors into the clear depths of Lake Baikal, turning 10 megatonnes of water into a neutrino detector. Another neutrino observatory, Antares (Astronomy with a Neutrino Telescope and Abyss Environmental Research), was built off the coast of Toulon in France in Mediterranean waters 2.5km deep. Antares complements IceCube as a northern hemisphere-based observatory.

Even with an instrument the size of IceCube, scientists expect to see only a few hundred neutrinos a day. The elusive particles reveal themselves on the rare occasions that they collide with the nuclei of oxygen atoms in the ice. When this happens, a neutrino produces a particle called a muon, a heavy relative of the electron. These muons travel faster than the speed of light in ice and release a shockwave of faint, blue light that is picked up by IceCube's light sensors.

In a lab on the surface, signals from DOMs throughout the IceCube observatory are combined and analysed to work out the direction and energy of neutrinos that left their tracks. The scientists will look for muons that move upwards through the ice, as these are produced by neutrinos that passed through the Earth before reaching the detector. Far more downwards-moving muons are produced by charged particles in the atmosphere above the detector, but these don't point back to the sources of cosmic rays. "We essentially use the Earth as a giant filter to absorb all the particle junk that is made locally," says Sarkar.

Over time, the IceCube observatory will build up a "neutrino map" of the sky and with luck find hotspots in the heavens where high-energy cosmic rays appear to come from. By comparing this map with those already made by optical, infra-red, radio and x-ray telescopes, scientists may finally learn where, and even how, cosmic rays are made.

Recently, the IceCube team signed an agreement with the Nasa scientists who operate the Swift satellite that scours space for gamma ray bursts, the most violent events in the universe. Whenever Swift spots one, Nasa tells the IceCube scientists so they can immediately check that part of the sky for neutrinos.

Klein says about 90% of the urge to understand cosmic rays is intellectual, but unravelling the natural processes that propel particles around in space could be used to transform technology on the ground. "If we can learn how cosmic rays are produced, we might learn something useful for building accelerators on Earth," he says.

One region of space that is a likely source of cosmic rays is a galaxy 10 million light years away called Centaurus A. There is little to see through an optical telescope because the galaxy is obscured by dust, but infrared images from Nasa's Spitzer space telescope cut through the haze to show a spiral galaxy falling into a black hole at the centre of Centaurus A. When Nasa's Chandra space telescope took x-ray images of Centaurus A, it saw huge jets erupting from the centre of the galaxy.

"Centaurus A looks different at every wavelength we've tried. The question is, what will it look like through a neutrino observatory?" says Sarkar. With neutrinos, scientists may finally be able to look deep into the heart of a galaxy and see what Sarkar calls the "central engine" that churns out cosmic rays.

Even as IceCube goes into action, scientists have begun work on prototype neutrino observatories that are larger still. The Arianna neutrino observatory will turn 100 cubic kilometres of the Ross ice shelf in Antarctica into a colossal neutrino detector.

Francis Halzen, IceCube's lead scientist at the University of Wisconsin-Madison, turns to Marcel Proust when asked how neutrino observatories such as IceCube might give us new insights into the workings of the cosmos: "The real voyage of discovery consists not in seeking new landscapes but in having new eyes."

As the short summer and its 24-hour days of sunlight come to an end at the south pole, work on IceCube has turned to upgrading computer systems and packing up the hot-water drill for long-term storage. Now the scientists face a waiting game: it is time to see if the Antarctic ice can catch their elusive quarry.