Science's hunt for a unifying account of how the world works requires us to entertain everything from hidden dimensions to multiple universes. But are these ideas based on fact or fiction? Jim Baggott and Mike Duff debate the limits of physics

Jim Baggott, author of Farewell to Reality: How Fairytale Physics Betrays the Search for Scientific Truth

The discovery of the Higgs boson was a triumph for the standard model of particle physics. This is the theory that describes reality at the level of elementary particles and the forces between them and which helps us to understand the nature of material substance. But we know the standard model can't be the whole story. There are lots of things it can't explain, such as the elementary particle masses, the existence of dark matter or dark energy, and it takes no account of the force of gravity. There are no clues in the available scientific data about how these problems might be solved, and theorists have been obliged to speculate. But, in Farewell to Reality, I argue that in their ambition to develop a "theory of everything", some theorists have crossed a line. The resulting theories, such as superstring theory (or M-theory), are not grounded in empirical data and produce no real predictions, so they can't be tested. Albert Einstein once warned: "Time and again the passion for understanding has led to the illusion that man is able to comprehend the objective world rationally by pure thought without any empirical foundations – in short, by metaphysics." Now, metaphysics is not science. Yet a string of recent bestselling popular science books, supported by press articles, radio and television documentaries, have helped to create the impression that this is all accepted scientific fact. Physics has gone too far.

Mike Duff, professor of theoretical physics at Imperial College London

It seems to me that you are conflating two different issues: (1) Are professional physicists barking up the wrong tree? (2) Is their work being misrepresented in the popular media? Let me respond to the first. The job of theoretical physicists is not only to explain what their experimental colleagues have discovered but also to predict phenomena that have not yet been found. Quantum theory, for example, was largely driven by empirical results, whereas Einstein's general theory of relativity was more a product of speculation and thought experiments (contrary to what your quote implies). Speculation, then, is a vital part of the scientific process. When Paul Dirac wrote down his equation describing how quantum particles behave he wasn't just explaining the electron, whose properties had been well established in experiments. His equation also predicted the hitherto undreamed-of positron, and hence the whole concept of antimatter. Such speculation is not a flight of fancy. It is always constrained by the straitjacket of mathematical consistency and compatibility with established laws. It is a common fallacy that physics is only about what has already been confirmed in experiments. The Higgs boson had no foundation in empirical reality when it was predicted in 1964.

JB These are indeed different issues, but they're closely related. Science is very forgiving in that nobody really cares how a new theory is arrived at, so long as it is a better theory. But, throughout history, even highly speculative theories have eventually been tested by reference to empirical facts. So, the positron was discovered in cosmic ray experiments just a couple of years after Dirac had agreed that this was what his theory predicted. The Higgs mechanism was invented in 1964 and used to predict the masses of other particles which were subsequently discovered at Cern in 1983. What looks like the Higgs boson was found last year.

Despite the best efforts of a community of more than 1,500 string theorists worldwide – efforts spanning more than four decades – there is still no single string theory prediction that allows a definitive test. For sure, there's lots of mathematical consistency and compatibility with established laws, but should a theory that makes no predictions be regarded as scientific? Aren't these really exercises in abstract mathematics? Or philosophy? At what point do we choose another tree to bark up?

And yes, this work is misrepresented in the popular media. The theorists themselves are misrepresenting it as accepted science.

MD Theories rarely spring fully formed from the minds of their discoverers. Chapter 2 of your book reminds us that it took 30 years of quantum entanglement (Einstein's "spooky action at a distance", proposed in 1935) before John Bell made a falsifiable prediction and another 20 before Alain Aspect tested it experimentally. Was all the entanglement research done in the meantime, including Einstein's, unscientific metaphysics? I don't think so.

Boson prediction: Peter Higgs.

String theory is not an unshakable edifice erected 40 years ago; it is an idea that is constantly undergoing modifications and improvements in the light of new discovery, for example the incorporation of membranes and M-theory in 1995. It provides a way (so far the only way) of reconciling gravity with quantum mechanics, including inter alia the microscopic explanation for Hawking's quantum black hole radiation: necessary steps towards the final theory, but not easily accessible to experiment. Definitive experimental tests will require that the theory also incorporate and improve upon the standard models of particle physics and cosmology. An impressive body of evidence in favour of this has accumulated, but it is still work in progress. Rest assured that, if anyone found another more promising tree, the 1,500 would start barking up that one.

As for misrepresentation in the media, there will always be sensationalists and attention-seekers in any field, but in my (admittedly biased) opinion, the worst culprits are the journalists.

JB You make an important point, but this is a bit of a minefield and we need to tread carefully. I'll admit that it's actually not possible to do science completely free of metaphysics. Any application of theory relies on things that we must accept as contingently true without proof. I like to think of this metaphysics as a kind of oil that lubricates the mix of ideas and data, allowing them to work together to provide genuine insight and understanding. In 1935, Einstein challenged the prevailing interpretation of quantum theory. This was indeed an exercise in pure metaphysics.

Theory of relativity: Albert Einstein.

But quantum theory was an already well-established, empirically verified structure. Einstein was tinkering with the oil, in the belief that how we seek to interpret quantum theory can have important practical consequences. All the ingredients were there, so I have no problem accepting this as perfectly legitimate science. My point is that in contemporary string theory a key ingredient is missing. There simply is no empirical data. Instead we have lots of ideas and lots of picturesque interpretation. The "evidence" you refer to is largely derived from relationships that can be established between these ideas but for which there is no factual support. Hawking radiation is no doubt a great idea, but we actually have no observational evidence for it. In other words, in string theory the metaphysics is all there is.

MD String theory is also built upon already well-established, empirically verified structures, namely general relativity and quantum field theory. Your entire case rests upon the claim that unless physicists are making falsifiable predictions rapidly tested by experiment they are in fairy land. But when confronted with a historical counterexample (involving Einstein, the scientist you most like to contrast with today's "fairytale" physicists) you try to wriggle out of it. Quantum entanglement is a real effect not metaphysics, but it took 50 years before the theoretical idea came to fruition. Yet you demand stricter deadlines for a theory of everything. Other examples of theoretical ideas with long gestation periods are not hard to find: atoms (400BC), black holes (1784), gravitational waves (1916). The W and Z bosons of the standard model of particle physics are described by the 1954 gauge theory of Yang and Mills, but they didn't know this at the time.

Falsifiable predictions concerning W and Z had to wait until 1969 and their experimental discovery until 1983.

Black hole inkling: Stephen Hawking.

By the way, the techniques of string theory have found application in other branches of physics such as quark-gluon plasmas, superconductivity, fluid mechanics and quantum information theory (as well as having a profound influence on pure mathematics). These serendipitous discoveries, not in the original string manifesto, are amenable to experimental test in the shorter term but would never have happened if the stop-string-theory critics had had their way.

JB All the examples you cite involved the interplay of ideas with observation and experiment. I don't dispute that some ideas took a long time to gain acceptance (although I think you'll find that in 400BC the Greeks were actually doing philosophy).

String theory breaks with this tradition. Yes, the maths provides hints of a theory of everything, but this comes at a big cost. We must assume that elementary particles are strings or membranes. We assume a "supersymmetry" between different types of particle. We assume that the theory's six extra spatial dimensions are compactified in a space so small we can never experience them. We assume that the five different string theories are subsumed in an overarching M-theory, but we don't know what this is. Because there are 10-to-the-power-500 different ways of compactifying the extra dimensions we assume that each of these describes a different type of universe in a multiverse of possibilities. Finally, we assume that the universe is the way it is because this is the only universe compatible with our existence.

There is no empirical evidence to support any of these assumptions. It's not surprising that the theory struggles to make any testable predictions. Recognising that they have a problem, some theorists are seeking to change the definition of science to accommodate this kind of metaphysics. I think this is very dangerous.

MD Yours is a common fallacy. Dirac did not assume the positron; he discovered it to be a consequence of an equation that described the well-established electron. Similarly, string theorists did not assume supersymmetry, extra dimensions, the dualities of M-theory or the myriad possible universes; they discovered them to be consequences of a theory that subsumes empirically well-established features such as general relativity, gauge field theory and chiral quarks and leptons. Current research is devoted to finding out what else M-theory requires.

Quantum pioneer: Paul Dirac

Moreover, there is a feeling, hard to convey to the layman but shared by many experienced theorists, that these ideas all hang together. As Peter Higgs said recently, "I'm a big fan of supersymmetry because it seems the only way to get gravity into the game''.

Finally, you offer no credible alternative. If you don't like string theory the answer is simple: come up with a better one. The battle for the correct theory will not be won on Amazon or on the blogosphere, however. It will be won in the pages of scholarly scientific journals. Sadly, many critics of string theory, having lost their case in the court of science, try to win it in the court of popular opinion. A science writer calling the theorists who are actually doing the research "confidence tricksters'' or Stephen Hawking "a fairytale physicist'' doesn't cut the mustard.