I’m sitting at my desk at the University of Washington trying to conserve energy. It isn’t me who’s losing it; it’s my computer simulations. Actually, colleagues down the hall might say I was losing it as well. When I tell people I’m working on speculative theories about dark matter, they start to speculate about me. I don’t think everyone who works in the building even believes in it.

In presentations, I point out how many cosmological puzzles it helps to solve. Occam’s Razor is my silver bullet: the fact that just one posit can explain so much. Then I talk about the things that standard dark matter doesn’t fix. There don’t seem to be enough satellite galaxies around our Milky Way. The inner shapes of small galaxies are inconsistent. I invoke Occam’s Razor again and argue that you can resolve these issues by adding a weak self-interaction to standard dark matter, a feeble scattering pattern when its particles collide. Then someone will ask me if I really believe in all this stuff. Tough question.

The world we see is an illusion, albeit a highly persistent one. We have gradually got used to the idea that nature’s true reality is one of uncertain quantum fields; that what we see is not necessarily what is. Dark matter is a profound extension of this concept. It appears that the majority of matter in the universe has been hidden from us. That puts physicists and the general public alike in an uneasy place. Physicists worry that they can’t point to an unequivocal confirmed prediction or a positive detection of the stuff itself. The wider audience finds it hard to accept something that is necessarily so shadowy and elusive. The situation, in fact, bears an ominous resemblance to the aether controversy of more than a century ago.

In the late-1800s, scientists were puzzled at how electromagnetic waves (for instance, light) could pass through vacuums. Just as the most familiar sort of waves are constrained to water — it’s the water that does the waving — it seemed obvious that there had to be some medium in which electromagnetic waves were ripples. Hence the notion of ‘aether’, an imperceptible field that was thought to permeate all of space.

The American scientists Albert Michelson and Edward Morley carried out the most famous experiment to probe the existence of aether in 1887. If light needed a medium to propagate, they reasoned, then the Earth ought to be moving through this same medium. They set up an ingenious apparatus to test the idea: a rigid optics table floating on a cushioning vat of liquid mercury such that the table could rotate in any direction. The plan was to compare the wavelengths of light beams travelling in different relative directions, as the apparatus rotated or as the Earth swung around the sun. As our planet travelled along its orbit in an opposite direction to the background aether, light beams should be impeded, compressing their wavelength. Six months later, the direction of the impedance should reverse and the wavelength would expand. But to the surprise of many, the wavelengths were the same no matter what direction the beams travelled in. There was no sign of the expected medium. Aether appeared to be a mistake.

This didn’t rule out its existence in every physicist’s opinion. Disagreement about the question rumbled on until at least some of the aether proponents died. Morley himself didn’t believe his own results. Only with perfect hindsight is the Michelson-Morley experiment seen as evidence for the absence of aether and, as it turned out, confirmation of Albert Einstein’s more radical theory of relativity.

Dark matter, dark energy, dark money, dark markets, dark biomass, dark lexicon, dark genome: scientists seem to add dark to any influential phenomenon that is poorly understood and somehow obscured from direct perception. The darkness, in other words, is metaphorical. At first, however, it was intended quite literally. In the 1930s, the Swiss astronomer Fritz Zwicky observed a cluster of galaxies, all gravitationally bound to each other and orbiting one another much too fast. Only the gravitational pull of a very large, unseen mass seemed capable of explaining why they did not simply spin apart. Zwicky postulated the presence of some kind of ‘dark’ matter in the most casual sense possible: he just thought there was something he couldn’t see. But astronomers have continued to find the signature of unseen mass throughout the cosmos. For example, the stars of galaxies also rotate too fast. In fact, it looks as if dark matter is the commonest form of matter in our universe.

It is also the most elusive. It does not interact strongly with itself or with the regular matter found in stars, planets or us. Its presence is inferred purely through its gravitational effects, and gravity, vexingly, is the weakest of the fundamental forces. But gravity is the only significant long-range force, which is why dark matter dominates the universe’s architecture at the largest scales.

Astrophysicists who try to trifle with the fundamentals of dark matter tend to find themselves cut off from the mainstream

In the past half-century, we have developed a standard model of cosmology that describes our observed universe quite well. In the beginning, a hot Big Bang caused a rapid expansion of space and sowed the seeds for fluctuations in the density of matter throughout the universe. Over the next 13.7 billion years, those density patterns were scaled up thanks to the relentless force of gravity, ultimately forming the cosmic scaffolding of dark matter whose gravitational pull suspends the luminous galaxies we can see.

This standard model of cosmology is supported by a lot of data, including the pervasive radiation field of the universe, the distribution of galaxies in the sky, and colliding clusters of galaxies. These robust observations combine expertise and independent analysis from many fields of astronomy. All are in strong agreement with a cosmological model that includes dark matter. Astrophysicists who try to trifle with the fundamentals of dark matter tend to find themselves cut off from the mainstream. It isn’t that anybody thinks it makes for an especially beautiful theory; it’s just that no other consistent, predictively successful alternative exists. But none of this explains what dark matter actually is. That really is a great, unsolved problem in physics.

So the hunt is on. Particle accelerators sift through data, detectors wait patiently underground, and telescopes strain upwards. The current generation of experiments has already placed strong constraints on viable theories. Optimistically, the nature of dark matter could be understood within a few decades. Pessimistically, it might never be understood.

We are in an era of discovery. A body of well-confirmed theory governs the assortment of fundamental particles that we have already observed. The same theory allows the existence of other, hitherto undetected particles. A few decades ago, theorists realised that a so-called Weakly Interacting Massive Particle (WIMP) might exist. This generic particle would have all the right characteristics to be dark matter, and it would be able to hide right under our noses. If dark matter is indeed a WIMP, it would interact so feebly with regular matter that we would have been able to detect it only with the generation of dark matter experiments that are just now coming on stream. The most promising might be the Large Underground Xenon (LUX) experiment in South Dakota, the biggest dark matter detector in the world. The facility opened in a former gold mine this February and is receptive to the most elusive of subatomic particles. And yet, despite LUX’s exquisite sensitivity, the hunt for dark matter itself has been something of a waiting game. So far, the only particles to turn up in the detector’s trap are bits of cosmic noise: nothing more than a nuisance.

The past success of standard paradigms in theoretical physics leads us to hunt for a single generic dark matter particle — the dark matter. Arguably, though, we have little justification for supposing that there is anything to be found at all; as the English physicist John D Barrow said in 1994: ‘There is no reason that the universe should be designed for our convenience.’ With that caveat in mind, it appears the possibilities are as follows. Either dark matter exists or it doesn’t. If it exists, then either we can detect it or we can’t. If it doesn’t exist, either we can show that it doesn’t exist or we can’t. The observations that led astronomers to posit dark matter in the first place seem too robust to dismiss, so the most common argument for non-existence is to say there must be something wrong with our understanding of gravity – that it must not behave as Einstein predicted. That would be a drastic change in our understanding of physics, so not many people want to go there. On the other hand, if dark matter exists and we can’t detect it, that would put us in a very inconvenient position indeed.

But we are living through a golden age of cosmology. In the past two decades, we have discovered so much: we have measured variations in the relic radiation of the Big Bang, learnt that the universe’s expansion is accelerating, glimpsed black holes and spotted the brightest explosions ever in the universe. In the next decades, we are likely to observe the first stars in the universe, map nearly the entire distribution of matter, and hear the cataclysmic merging of black holes through gravitational waves. Even among these riches, dark matter offers a uniquely inviting prospect, sitting at a confluence of new observations, theory, technology and (we hope) new funding.

Physicists could take non-detection as a hint to give up, but there is always the teasing possibility that we just need a better experiment

The various proposals to get its measure tend to fall into one of three categories: artificial creation (in a particle accelerator), indirect detection, and direct detection. The last, in which researchers attempt to catch WIMPs in the wild, is where the excitement is. The underground LUX detector is one of the first in a new generation of ultra-sensitive experiments. It counts on the WIMP interacting with the nucleus of a regular atom. These experiments generally consist of a very pure detector target, such as pristine elemental Germanium or Xenon, cooled to extremely low temperatures and shielded from outside particles. The problem is that stray particles tend to sneak in anyway. Interloper interactions are carefully monitored. Noise reduction, shielding and careful statistics are the only way to confirm real dark-matter interaction events from false alarms.

Theorists have considered a lot of possibilities for how the real thing might work with the standard WIMP. Actually, the first generation of experiments has already ruled out the so-called z-boson scattering interaction. What is left is Higgs boson-mediated scattering, which would involve the same particle that the Large Hadron Collider discovered in Geneva in November last year. That implies a very weak interaction, but it would be perfectly matched to the current sensitivity threshold of the new generation of experiments.

Then again, science is less about saying what is than what is not, and non-detections have placed relatively interesting constraints on dark matter. They have also, in a development that is strikingly reminiscent of the aether controversy, thrown out some anomalies that need to be cleared up. Using a different detector target to LUX, the Italian DAMA (short for ‘DArk MAtter’) experiment claims to have found an annual modulation of their dark matter signal. Detractors dispute whether they really have any signal at all. Just like with the aether, we expected to see this kind of yearly variation, as the Earth orbits the Sun, sometimes moving with the larger galactic rotation and sometimes against it. The DAMA collaboration measured such an annual modulation. Other competing projects (XENON, CDMS, Edelweiss and ZEPLIN, for example) didn’t, but these experiments cannot be compared directly, so we should probably reserve judgment.

Nature can be cruel. Physicists could take non-detection as a hint to give up, but there is always the teasing possibility that we just need a better experiment. Or perhaps dark matter will reveal itself to be almost as complex as regular matter. Previous experiments imposed quite strict limitations on just how much complexity we can expect — there’s no prospect of dark-matter people, or even dark-matter chemistry, really — but it could still come in multiple varieties. We might find a kind of particle that explains only a fraction of the expected total mass of dark matter.

In a sense, this has already occurred. Neutrinos are elusive but widespread (60 billion of them pass through an area the size of your pinky every second). They hardly ever interact with regular matter, and until 1998 we thought they were entirely massless. In fact, neutrinos make up a tiny fraction of the mass budget of the universe, and they do act like an odd kind of dark matter. They aren’t ‘the’ dark matter, but perhaps there is no single type of dark matter to find.

To say that we are in an era of discovery is really just to say that we are in an era of intense interest. Physicists say we would have achieved something if we determine that dark matter is not a WIMP. Would that not be a discovery? At the same time, the field is burgeoning with ideas and rival theories. Some are exploring the idea that dark matter has interactions, but we will never be privy to them. In this scenario, dark matter would have an interaction at the smallest of scales which would leave standard cosmology unchanged. It might even have an exotic universe of its own: a dark sector. This possibility is at once terrifying and entrancing to physicists. We could posit an intricate dark matter realm that will always escape our scrutiny, save for its interaction with our own world through gravity. The dark sector would be akin to a parallel universe.

It is rather easy to tinker with the basic idea of dark matter when you make all of your modifications very feeble. And so this is what all dark matter theorists are doing. I have run with the idea that dark matter might have self-interactions and worked that into supercomputer simulations of galaxies. On the largest scales, where cosmology has made firm predictions, this modification does nothing, but on small scales, where the theory of dark matter shows signs of faltering, it helps with several issues. The simulations are pretty to look at and they make acceptable predictions. There are too many free parameters, though — what scientists call fine-tuning — such that the results can seem tailored to fit the observations. That’s why I reserve judgement, and you would be well advised to do the same.

We will probably never know for certain whether dark matter has self-interactions. At best, we might put an upper limit on how strong such interactions could be. So, when people ask me if I think self-interacting dark matter is the correct theory, I say no. I am constraining what is possible, not asserting what is. But this is kind of disappointing, isn’t it? Surely cosmology should hold some deep truth that we can hope to grasp.

One day, perhaps, LUX or one of its competitors might discover just what they are looking for. Or maybe on some unassuming supercomputer, I will uncover a hidden truth about dark matter. Regardless, such a discovery will feel removed from us, mediated as it will be through several layers of ghosts in machines. The dark matter universe is part of our universe, but it will never feel like our universe.

Nature plays an epistemological trick on us all. The things we observe each have one kind of existence, but the things we cannot observe could have limitless kinds of existence. A good theory should be just complex enough. Dark matter is the simplest solution to a complicated problem, not a complicated solution to simple problem. Yet there is no guarantee that it will ever be illuminated. And whether or not astrophysicists find it in a conceptual sense, we will never grasp it in our hands. It will remain out of touch. To live in a universe that is largely inaccessible is to live in a realm of endless possibilities, for better or worse.