The world's most advanced instrument for detecting dark matter – the invisible substance thought to account for most of the matter in the universe – has ended its first three months of service without catching a hint of the stuff.

Sat a mile underground in a defunct gold mine in South Dakota, the Large Underground Xenon (Lux) experiment aims to detect dark matter particles as they drift through the Earth. It is the most sensitive detector for the mysterious substance ever made.

The failure to even glimpse dark matter means that tentative sightings by other experiments are almost certainly false. The Lux results do not rule out the existence of dark matter, but instead narrow down the possible forms it might take.

"We saw nothing," said Richard Gaitskell, a co-spokesman for the Lux experiment at Brown University in Providence, Rhode Island. "We do not have a single dark matter candidate event."

The nature of dark matter has mystified scientists for 80 years, since the Swiss astronomer Fritz Zwicky noticed that galaxies rotated in such a way that they must contain more matter than was visible. His observations have been confirmed time and again in recent decades. Around 85% of the matter in the universe is now thought to be "dark", meaning it neither emits nor reflects light.

Measurements by the European Space Agency's Planck mission showed in March that normal matter – the stuff of stars, planets and people – accounts for less than 5% of the mass-energy in the observable universe, while dark matter makes up around 27%. The rest of the cosmos, around 68%, is labelled dark energy and is thought to be the driving force behind the expansion of the universe.

"We have entered the new millennium and yet we still have no idea what 95% of the universe is made of. Our level of ignorance is quite staggering, and it's one of the largest challenges we have right now," said Gaitskell.

At the centre of the Lux detector in the Sanford mine in South Dakota is a two-metre-high titanium tank filled with 370kg of liquid xenon cooled to -100C. The detector is housed far underground to shield it from energetic cosmic rays that can create spurious signals in the instrument. The detector tank is surrounded by pure, de-ionised water to provide more shielding.

It is thought dark matter usually passes straight through normal matter, but not always. If a dark matter particle hit a xenon atom in the Lux detector, it would produce a tiny flash of light that would be picked up by sensors at the top and bottom of the tank. The collision would produce electrons too, which would be pulled to the top of the tank by an electric field. Here, they would meet a thin layer of xenon gas and produce a second flash of light, used to confirm the event.

Results from the first three-month run of the LUX detector drew a blank on a family of dark matter particles called weakly interacting massive particles, or wimps. That rules out hints of dark matter reported by experiments such as the Cryogenic Dark Matter Search (CDMS) and Coherent Germanium Neutrino Technology (Cogent) detectors at the Soudan mine in Minnesota.

In April, scientists working on CDMS said they had tentative evidence for three dark matter particles. But if the signals were real, Lux would have seen more than 1,500 of the particles during its run. "We see absolutely no events, and that's a hell of a disagreement," said Gaitskell.

But Gaitskell remains optimistic. "You only become deflated if you view scientific problems in the context of individual scientists. That's not a useful way to see things when you are trying to make scientific discoveries. If you bind it up with individual egos, that's when mistakes are made," he said. "We are closing in on dark matter."

Henrique Araujo, a physicist on Lux based at Imperial College London, said the experiment will now be prepared for a longer, 300-day search for dark matter. The scientists will not only look for wimps, but also weakly interacting slim particles, or wisps.

One theory, called supersymmetry, predicts a particle called a neutralino that could be a component of dark matter. But confidence in supersymmetry has taken a beating because the Large Hadron Collider has found no evidence to support the theory. Another option is a member of the wisp family of particles called an axion. The arcane nature of physics calls for some imagination when it comes to naming particles. The axion was named after a washing powder.

"We don't really know what the universe is made of," Araujo told the Guardian. "We understand only a few percent of it, so we are frying the small fish before we are frying the big ones. Without dark matter we know very little about how the universe really evolved and what its fate is."

Physicists are already designing, and are nearly ready to build, an upgraded dark matter detector called LZ that will be 100 times more sensitive than Lux. The experiment will replace Lux in the Sanford mine and use a tank filled with seven tonnes of xenon.

"The overwhelming evidence from cosmology is there has to be something out there that is like dark matter, but that is the only statement we can make," said Pedro Ferreira, an astrophysicist at Oxford University. "All we know is there is this thing that clumps but we don't know anything about its properties. We have nothing to go on. If we could find the dark matter, our amount of information increases dramatically. From zero."