It is the biggest of problems, it is the smallest of problems. At present physicists have two separate rulebooks explaining how nature works. There is general relativity, which beautifully accounts for gravity and all of the things it dominates: orbiting planets, colliding galaxies, the dynamics of the expanding universe as a whole. That’s big. Then there is quantum mechanics, which handles the other three forces – electromagnetism and the two nuclear forces. Quantum theory is extremely adept at describing what happens when a uranium atom decays, or when individual particles of light hit a solar cell. That’s small.

Now for the problem: relativity and quantum mechanics are fundamentally different theories that have different formulations. It is not just a matter of scientific terminology; it is a clash of genuinely incompatible descriptions of reality.

The conflict between the two halves of physics has been brewing for more than a century – sparked by a pair of 1905 papers by Einstein, one outlining relativity and the other introducing the quantum – but recently it has entered an intriguing, unpredictable new phase. Two notable physicists have staked out extreme positions in their camps, conducting experiments that could finally settle which approach is paramount.

Basically you can think of the division between the relativity and quantum systems as “smooth” versus “chunky”. In general relativity, events are continuous and deterministic, meaning that every cause matches up to a specific, local effect. In quantum mechanics, events produced by the interaction of subatomic particles happen in jumps (yes, quantum leaps), with probabilistic rather than definite outcomes. Quantum rules allow connections forbidden by classical physics. This was demonstrated in a much-discussed recent experiment in which Dutch researchers defied the local effect. They showed that two particles – in this case, electrons – could influence each other instantly, even though they were a mile apart. When you try to interpret smooth relativistic laws in a chunky quantum style, or vice versa, things go dreadfully wrong.

Relativity gives nonsensical answers when you try to scale it down to quantum size, eventually descending to infinite values in its description of gravity. Likewise, quantum mechanics runs into serious trouble when you blow it up to cosmic dimensions. Quantum fields carry a certain amount of energy, even in seemingly empty space, and the amount of energy gets bigger as the fields get bigger. According to Einstein, energy and mass are equivalent (that’s the message of E=mc2), so piling up energy is exactly like piling up mass. Go big enough, and the amount of energy in the quantum fields becomes so great that it creates a black hole that causes the universe to fold in on itself. Oops.

‘Quantum mechanics provided the conceptual tools for the Large Hadron Collider.’ Photograph: Rex Features

Craig Hogan, a theoretical astrophysicist at the University of Chicago and the director of the Center for Particle Astrophysics at Fermilab, is reinterpreting the quantum side with a novel theory in which the quantum units of space itself might be large enough to be studied directly. Meanwhile, Lee Smolin, a founding member of the Perimeter Institute for Theoretical Physics in Waterloo, Canada, is seeking to push physics forward by returning to Einstein’s philosophical roots and extending them in an exciting direction.

To understand what is at stake, look back at the precedents. When Einstein unveiled general relativity, he not only superseded Isaac Newton’s theory of gravity; he also unleashed a new way of looking at physics that led to the modern conception of the Big Bang and black holes, not to mention atomic bombs and the time adjustments essential to your phone’s GPS. Likewise, quantum mechanics did much more than reformulate James Clerk Maxwell’s textbook equations of electricity, magnetism and light. It provided the conceptual tools for the Large Hadron Collider, solar cells, all of modern microelectronics.

What emerges from the dust-up could be nothing less than a third revolution in modern physics, with staggering implications. It could tell us where the laws of nature came from, and whether the cosmos is built on uncertainty or whether it is fundamentally deterministic, with every event linked definitively to a cause.

Small is beautiful

Hogan, champion of the quantum view, is what you might call a lamp-post physicist: rather than groping about in the dark, he prefers to focus his efforts where the light is bright, because that’s where you are most likely to be able to see something interesting. That’s the guiding principle behind his current research. The clash between relativity and quantum mechanics happens when you try to analyse what gravity is doing over extremely short distances, he notes, so he has decided to get a really good look at what is happening right there. “I’m betting there’s an experiment we can do that might be able to see something about what’s going on, about that interface that we still don’t understand,” he says.

A basic assumption in Einstein’s physics – an assumption going all the way back to Aristotle, really – is that space is continuous and infinitely divisible, so that any distance could be chopped up into even smaller distances. But Hogan questions whether that is really true. Just as a pixel is the smallest unit of an image on your screen and a photon is the smallest unit of light, he argues, so there might be an unbreakable smallest unit of distance: a quantum of space.

Chunky space does not neatly align with the ideas in string theory – or in any other proposed physics model

In Hogan’s scenario, it would be meaningless to ask how gravity behaves at distances smaller than a single chunk of space. There would be no way for gravity to function at the smallest scales because no such scale would exist. Or put another way, general relativity would be forced to make peace with quantum physics, because the space in which physicists measure the effects of relativity would itself be divided into unbreakable quantum units. The theatre of reality in which gravity acts would take place on a quantum stage.

Hogan acknowledges that his concept sounds a bit odd, even to a lot of his colleagues on the quantum side of things. Since the late 1960s, a group of physicists and mathematicians have been developing a framework called string theory to help reconcile general relativity with quantum mechanics; over the years, it has evolved into the default mainstream theory, even as it has failed to deliver on much of its early promise. Like the chunky-space solution, string theory assumes a fundamental structure to space, but from there the two diverge. String theory posits that every object in the universe consists of vibrating strings of energy. Like chunky space, string theory averts gravitational catastrophe by introducing a finite, smallest scale to the universe, although the unit strings are drastically smaller even than the spatial structures Hogan is trying to find.

Chunky space does not neatly align with the ideas in string theory – or in any other proposed physics model, for that matter. “It’s a new idea. It’s not in the textbooks; it’s not a prediction of any standard theory,” Hogan says, sounding not the least bit concerned. “But there isn’t any standard theory, right?”

If he is right about the chunkiness of space, that would knock out a lot of the current formulations of string theory and inspire a fresh approach to reformulating general relativity in quantum terms. It would suggest new ways to understand the inherent nature of space and time. And weirdest of all, perhaps, it would bolster the notion that our seemingly three-dimensional reality is composed of more basic, two-dimensional units. Hogan takes the “pixel” metaphor seriously: just as a TV picture can create the impression of depth from a bunch of flat pixels, he suggests, so space itself might emerge from a collection of elements that act as if they inhabit only two dimensions.

Like many ideas from the far edge of today’s theoretical physics, Hogan’s speculations can sound suspiciously like late-night philosophising in the freshman dorm. What makes them drastically different is that he plans to put them to a hard experimental test. As in, right now.

Starting in 2007, Hogan began thinking about how to build a device that could measure the exceedingly fine graininess of space. As it turns out, his colleagues had plenty of ideas about how to do that, drawing on technology developed to search for gravitational waves. Within two years Hogan had put together a proposal and was working with collaborators at Fermilab, the University of Chicago and other institutions to build a chunk-detecting machine, which he more elegantly calls a “holometer”. (The name is an esoteric pun, referencing both a 17th-century surveying instrument and the theory that 2D space could appear three-dimensional, analogous to a hologram.)

Beneath its layers of conceptual complexity, the holometer is technologically little more than a laser beam, a half-reflective mirror to split the laser into two perpendicular beams, and two other mirrors to bounce those beams back along a pair of 40m-long tunnels. The beams are calibrated to register the precise locations of the mirrors. If space is chunky, the locations of the mirrors would constantly wander about (strictly speaking, space itself is doing the wandering), creating a constant, random variation in their separation. When the two beams are recombined, they’d be slightly out of sync, and the amount of the discrepancy would reveal the scale of the chunks of space.

For the scale of chunkiness that Hogan hopes to find, he needs to measure distances to an accuracy of 10-18m, about 100m times smaller than a hydrogen atom, and collect data at a rate of about 100m readings per second. Amazingly, such an experiment is not only possible, but practical. “We were able to do it pretty cheaply because of advances in photonics, a lot of off-the-shelf parts, fast electronics and things like that,” Hogan says. “It’s a pretty speculative experiment, so you wouldn’t have done it unless it was cheap.” The holometer is currently humming away, collecting data at the target accuracy; he expects to have preliminary readings by the end of the year.

Hogan has his share of fierce sceptics, including many within the theoretical physics community. The reason for the disagreement is easy to appreciate: a success for the holometer would mean failure for a lot of the work being done in string theory. Despite this superficial sparring, though, Hogan and most of his theorist colleagues share a deep core conviction: they broadly agree that general relativity will ultimately prove subordinate to quantum mechanics. The other three laws of physics follow quantum rules, so it makes sense that gravity must as well.

For most of today’s theorists, however, belief in the primacy of quantum mechanics runs deeper still. At a philosophical – epistemological – level, they regard the large-scale reality of classical physics as a kind of illusion, an approximation that emerges from the more “true” aspects of the quantum world operating at an extremely small scale. Chunky space certainly aligns with that worldview.

Hogan likens his project to the landmark Michelson-Morley experiment of the 19th century, which searched for the aether – the hypothetical substance of space that, according to the leading theory of the time, transmitted light waves through a vacuum. The experiment found nothing; that perplexing null result helped inspire Einstein’s special theory of relativity, which in turn spawned the general theory of relativity and eventually turned the entire world of physics upside down. Adding to the historical connection, the Michelson-Morley experiment also measured the structure of space using mirrors and a split beam of light, following a setup remarkably similar to Hogan’s.

“We’re doing the holometer in that kind of spirit. If we don’t see something or we do see something, either way it’s interesting. The reason to do the experiment is just to see whether we can find something to guide the theory,” Hogan says. “You find out what your theorist colleagues are made of by how they react to this idea. There’s a world of very mathematical thinking out there. I’m hoping for an experimental result that forces people to focus the theoretical thinking in a different direction.”

Whether or not he finds his quantum structure of space, Hogan is confident the holometer will help physics address its big-small problem. It will show the right way (or rule out the wrong way) to understand the underlying quantum structure of space and how that affects the relativistic laws of gravity flowing through it.

A bigger vision

If you are looking for a totally different direction, Smolin of the Perimeter Institute is your man. Where Hogan goes gently against the grain, Smolin is a full-on dissenter: “There’s a thing that Richard Feynman told me when I was a graduate student. He said, approximately, ‘If all your colleagues have tried to demonstrate that something’s true and failed, it might be because that thing is not true.’ Well, string theory has been going for 40 or 50 years without definitive progress.”

And that is just the start of a broader critique. Smolin thinks the small-scale approach to physics is inherently incomplete. Current versions of quantum field theory do a fine job explaining how individual particles or small systems of particles behave, but they fail to take into account what is needed to have a sensible theory of the cosmos as a whole. They don’t explain why reality is like this, and not like something else. In Smolin’s terms, quantum mechanics is merely “a theory of subsystems of the universe”.

A more fruitful path forward, he suggests, is to consider the universe as a single enormous system, and to build a new kind of theory that can apply to the whole thing. And we already have a theory that provides a framework for that approach: general relativity. Unlike the quantum framework, general relativity allows no place for an outside observer or external clock, because there is no “outside”. Instead, all of reality is described in terms of relationships between objects and between different regions of space. Even something as basic as inertia (the resistance of your car to move until forced to by the engine, and its tendency to keep moving after you take your foot off the accelerator) can be thought of as connected to the gravitational field of every other particle in the universe.

‘What if the universe were entirely empty except for two astronauts, one of them spinning, the other stationary?’ Photograph: Nasa/Reuters

That last statement is strange enough that it’s worth pausing for a moment to consider it more closely. Consider a thought problem, closely related to the one that originally led Einstein to this idea in 1907. What if the universe were entirely empty except for two astronauts? One of them is spinning, the other is stationary. The spinning one feels dizzy, doing cartwheels in space. But which one of the two is spinning? From either astronaut’s perspective, the other is the one spinning. Without any external reference, Einstein argued, there is no way to say which one is correct, and no reason why one should feel an effect different from what the other experiences.

The distinction between the two astronauts makes sense only when you reintroduce the rest of the universe. In the classic interpretation of general relativity, then, inertia exists only because you can measure it against the entire cosmic gravitational field. What holds true in that thought problem holds true for every object in the real world: the behaviour of each part is inextricably related to that of every other part. If you’ve ever felt as if you wanted to be a part of something big, well, this is the right kind of physics for you. It is also, Smolin thinks, a promising way to obtain bigger answers about how nature really works, across all scales.

“General relativity is not a description of subsystems. It is a description of the whole universe as a closed system,” he says. When physicists are trying to resolve the clash between relativity and quantum mechanics, therefore, it seems like a smart strategy for them to follow Einstein’s lead and go as big as they possibly can.

Smolin is keenly aware that he is pushing against the prevailing devotion to small-scale, quantum-style thinking. “I don’t mean to stir things up; it just kind of happens that way. My role is to think clearly about these difficult issues, put my conclusions out there, and let the dust settle,” he says genially. “I hope people will engage with the arguments, but I really hope that the arguments lead to testable predictions.”

At first blush, Smolin’s ideas sound like a formidable starting point for concrete experimentation. Much as all of the parts of the universe are linked across space, they may also be linked across time, he suggests. His arguments led him to hypothesise that the laws of physics evolve over the history of the universe. Over the years, he has developed two detailed proposals for how this might happen. His theory of cosmological natural selection, which he hammered out in the 1990s, envisions black holes as cosmic eggs that hatch new universes. More recently, he has developed a provocative hypothesis about the emergence of the laws of quantum mechanics, called the principle of precedence – and this one seems much more readily put to the test.

Smolin’s principle of precedence arises as an answer to the question of why physical phenomena are reproducible. If you perform an experiment that has been performed before, you expect the outcome will be the same as in the past. (Strike a match and it bursts into flame; strike another match the same way and… you get the idea.) Reproducibility is such a familiar part of life that we typically don’t even think about it. We simply attribute consistent outcomes to the action of a natural “law” that acts the same way at all times. Smolin hypothesises that those laws actually may emerge over time, as quantum systems copy the behaviour of similar systems in the past.

One possible way to catch emergence in the act is by running an experiment that has never been done before, so there is no past version (that is, no precedent) for it to copy. Such an experiment might involve the creation of a highly complex quantum system, containing many components that exist in a novel entangled state. If the principle of precedence is correct, the initial response of the system will be essentially random. As the experiment is repeated, however, precedence builds up and the response should become predictable… in theory. “A system by which the universe is building up precedent would be hard to distinguish from the noises of experimental practice,” Smolin concedes, “but it’s not impossible.”

Although precedence can play out at the atomic scale, its influence would be system-wide, cosmic. It ties back to Smolin’s idea that small-scale, reductionist thinking seems like the wrong way to solve the big puzzles. Getting the two classes of physics theories to work together, though important, is not enough, either. What he wants to know – what we all want to know – is why the universe is the way it is. Why does time move forward and not backward? How did we end up here, with these laws and this universe, not some others?

The present lack of any meaningful answer to those questions reveals “something deeply wrong with our understanding of quantum field theory”, Smolin says. Like Hogan, he is less concerned about the outcome of any one experiment than he is with the larger programme of seeking fundamental truths. For Smolin, that means being able to tell a complete, coherent story about the universe; it means being able to predict experiments, but also to explain the unique properties that made atoms, planets, rainbows and people. Here again he draws inspiration from Einstein.

“The lesson of general relativity, again and again, is the triumph of relationalism,” Smolin says. The most likely way to get the big answers is to engage with the universe as a whole.

And the winner is?

If you wanted to pick a referee in the big-small debate, you could hardly do better than Sean Carroll, an expert in cosmology, field theory and gravitational physics at Caltech. He knows his way around relativity, he knows his way around quantum mechanics, and he has a healthy sense of the absurd: he calls his personal blog Preposterous Universe. Right off the bat, Carroll awards most of the points to the quantum side. “Most of us in this game believe that quantum mechanics is much more fundamental than general relativity is,” he says. That has been the prevailing view ever since the 1920s, when Einstein tried and repeatedly failed to find flaws in the counterintuitive predictions of quantum theory. The recent Dutch experiment demonstrating an instantaneous quantum connection between two widely separated particles – the kind of event that Einstein derided as “spooky action at a distance” – only underscores the strength of the evidence.

Illustration by Owen Gildersleeve

Taking a larger view, the real issue is not general relativity versus quantum field theory, Carroll explains, but classical dynamics versus quantum dynamics. Relativity, despite its perceived strangeness, is classical in how it regards cause and effect; quantum mechanics most definitely is not. Einstein was optimistic that some deeper discoveries would uncover a classical, deterministic reality hiding beneath quantum mechanics, but no such order has yet been found. The demonstrated reality of spooky action at a distance argues that such order does not exist.

“If anything, people underappreciate the extent to which quantum mechanics just completely throws away our notions of space and locality [the notion that a physical event can affect only its immediate surroundings]. Those things simply are not there in quantum mechanics,” Carroll says. They may be large-scale impressions that emerge from very different small-scale phenomena, like Hogan’s argument about 3D reality emerging from 2D quantum units of space.

Despite that seeming endorsement, Carroll regards Hogan’s holometer as a long shot, though he admits it is removed from his area of research. At the other end, he doesn’t think much of Smolin’s efforts to start with space as a fundamental thing; he believes the notion is as absurd as trying to argue that air is more fundamental than atoms. As for what kind of quantum system might take physics to the next level, Carroll remains broadly optimistic about string theory, which he says “seems to be a very natural extension of quantum field theory”. In all these ways, he is true to the mainstream, quantum-based thinking in modern physics.

Yet Carroll’s ruling, while almost entirely pro-quantum, is not purely an endorsement of small-scale thinking. There are still huge gaps in what quantum theory can explain. “Our inability to figure out the correct version of quantum mechanics is embarrassing,” he says. “And our current way of thinking about quantum mechanics is simply a complete failure when you try to think about cosmology or the whole universe. We don’t even know what time is.” Both Hogan and Smolin endorse this sentiment, although they disagree about what to do in response. Carroll favours a bottom-up explanation in which time emerges from small-scale quantum interactions, but declares himself “entirely agnostic” about Smolin’s competing suggestion that time is more universal and fundamental. In the case of time, then, the jury is still out.

No matter how the theories shake out, the large scale is inescapably important, because it is the world we inhabit and observe. In essence, the universe as a whole is the answer, and the challenge to physicists is to find ways to make it pop out of their equations. Even if Hogan is right, his space-chunks have to average out to the smooth reality we experience every day. Even if Smolin is wrong, there is an entire cosmos out there with unique properties that need to be explained – something that, for now at least, quantum physics alone cannot do.

By pushing at the bounds of understanding, Hogan and Smolin are helping the field of physics make that connection. They are nudging it toward reconciliation not just between quantum mechanics and general relativity, but between idea and perception. The next great theory of physics will undoubtedly lead to beautiful new mathematics and unimaginable new technologies. But the best thing it can do is create deeper meaning that connects back to us, the observers, who get to define ourselves as the fundamental scale of the universe.

• This essay originally appeared in issue 29 of Nautilus. To find out more, visit nautil.us

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.