Who doesn’t like a pretty idea? Physicists certainly do. In the foundations of physics, it has become accepted practice to prefer hypotheses that are aesthetically pleasing. Physicists believe that their motivations don’t matter because hypotheses, after all, must be tested. But most of their beautiful ideas are hard or impossible to test. And whenever an experiment comes back empty-handed, physicists can amend their theories to accommodate the null results.

This has been going on for about 40 years. In these 40 years, aesthetic arguments have flourished into research programmes – such as supersymmetry, the multiverse and grand unification – that now occupy thousands of scientists. In these 40 years, society spent billions of dollars on experiments that found no evidence to support the beautiful ideas. And in these 40 years, there has not been a major breakthrough in the foundations of physics.

My colleagues argue that criteria of beauty are experience-based. The most fundamental theories we currently have – the standard model of particle physics and Albert Einstein’s general relativity – are beautiful in specific ways. I agree it was worth a try to assume that more fundamental theories are beautiful in similar ways. But, well, we tried, and it didn’t work. Nevertheless, physicists continue to select theories based on the same three criteria of beauty: simplicity, naturalness, and elegance.

With simplicity I don’t mean Occam’s razor, which demands that among two theories that achieve the same thing, you pick the one that’s simpler. No, I mean absolute simplicity: a theory should be simple, period. When theories are not simple enough for my colleagues’ tastes, they try to make them simpler – by unifying several forces or by postulating new symmetries that combine particles in orderly sets.

The second criterion is naturalness. Naturalness is an attempt to get rid of the human element by requiring that a theory should not use assumptions that appear hand-picked. This criterion is most often applied to the values of constants without units, such as the ratios of elementary particles’ masses. Naturalness demands that such numbers should be close to one or, if that’s not the case, the theory explains why that isn’t so.

Then there’s elegance, the third and most elusive aspect of beauty. It’s often described as a combination of simplicity and surprise that, taken together, reveals new connections. We find elegance in the ‘Aha effect’, the moment of insight when things fall into place.

Physicists currently consider a theory promising if it’s beautiful according to these three criteria. This led them to predict, for example, that protons should be able to decay. Experiments have looked for this since the 1980s, but so far nobody has seen a proton decay. Theorists also predicted that we should be able to detect dark matter particles, such as axions or weakly interacting massive particles (WIMPs). We have commissioned dozens of experiments but haven’t found any of the hypothetical particles – at least not so far. The same criteria of symmetry and naturalness led many physicists to believe that the Large Hadron Collider (LHC) should see something new besides the Higgs boson, for example so-called ‘supersymmetric’ particles or additional dimensions of space. But none have been found so far.

How far can you push this programme before it becomes absurd? Well, if you make a theory simpler and simpler it will eventually become unpredictive, because the theory no longer contains enough information to even carry through calculations. What you get then is what theorists now call a ‘multiverse’ – an infinite collection of universes with different laws of nature.

For example, if you use the law of gravity without fixing the value of Newton’s constant by measurement, you could say that your theory contains a universe for any value of the constant. Of course, you then have to postulate that we live in the one universe that has the value of Newton’s constant that we happen to measure. So it might look like you haven’t gained much. Except that theorists can now write papers about that large number of new universes. Even better, the other universes aren’t observable, hence multiverse theories are safe from experimental test.

I think it’s time we take a lesson from the history of science. Beauty does not have a good track record as a guide for theory-development. Many beautiful hypotheses were just wrong, like Johannes Kepler’s idea that planetary orbits are stacked in regular polyhedrons known as ‘Platonic solids’, or that atoms are knots in an invisible aether, or that the Universe is in a ‘steady state’ rather than undergoing expansion.

And other theories that were once considered ugly have stood the test of time. When Kepler suggested that the planets move on ellipses rather than circles, that struck his contemporaries as too ugly to be true. And the physicist James Maxwell balked at his own theory involving electric and magnetic fields, because in his day the beauty standard involved gears and bolts. Paul Dirac chided a later version of Maxwell’s theory as ugly, because it required complicated mathematical gymnastics to remove infinities. Nevertheless, those supposedly ugly ideas were correct. They are still in use today. And we no longer find them ugly.

History has a second lesson. Even though beauty was arguably a strong personal motivator for many physicists, the problems that led to breakthroughs were not merely aesthetic misgivings – they were mathematical contradictions. Einstein, for example, abolished absolute time because it was in contradiction with Maxwell’s electromagnetism, thereby creating special relativity. He then resolved the conflict between special relativity and Newtonian gravity, which gave him general relativity. Dirac later removed the disagreement between special relativity and quantum mechanics, which led to the development of the quantum field theories which we still use in particle physics today.

The Higgs boson, too, was born out of need for logical consistency. Found at the LHC in 2012, the Higgs boson is necessary to make the standard model work. Without the Higgs, particle physicists’ calculations return probabilities larger than 1, mathematical nonsense that cannot describe reality. Granted, the mathematics didn’t tell us it had to be the Higgs boson, it could have been something else. But we knew that something new had to happen at the LHC, before it was even built. This was reasoning built on solid mathematical ground.

Supersymmetric particles, on the other hand, are pretty but not necessary. They were included to fix an aesthetic shortcoming of the current theory, a lack of naturalness. There’s nothing mathematically wrong with a theory that’s not supersymmetric, it’s just not particularly pretty. Particle physicists used supersymmetry to remedy this perceived shortfall, thereby making the theory much more beautiful. The predictions that supersymmetric particles should be seen at the LHC, therefore, were based on hope rather than sound logic. And the particles have not been found.

My conclusion from this long line of null results is that when physics tries to rectify a perceived lack of beauty, we waste time on problems that aren’t really problems. Physicists must rethink their methods, now – before we start discussing whether the world needs a next larger particle collider or yet another dark matter search.

The answer can’t be that anything goes, of course. The idea that new theories should solve existing problems is good in principle – it’s just that, currently, the problems themselves aren’t sharply formulated enough for that criterion to be useful. The conceptual and philosophical basis of reasoning in the foundations of physics is weak, and this must improve.

It’s no use, and not good scientific practice, to demand that nature conform to our ideals of beauty. We should let evidence lead the way to new laws of nature. I am pretty sure beauty will await us there.

Sabine Hossenfelder will be discussing elegant equations, simple laws and her new book Lost in Math: How Beauty Leads Physics Astray at this year’s HowTheLightGetsIn festival, a two-day philosophy and music festival in London, September 2018.