Albert Einstein’s theory of general relativity is a century old next year and, as far as the test of time is concerned, it seems to have done rather well. For many, indeed, it doesn’t merely hold up: it is the archetype for what a scientific theory should look like. Einstein’s achievement was to explain gravity as a geometric phenomenon: a force that results from the distortion of space-time by matter and energy, compelling objects – and light itself – to move along particular paths, very much as rivers are constrained by the topography of their landscape. General relativity departs from classical Newtonian mechanics and from ordinary intuition alike, but its predictions have been verified countless times. In short, it is the business.

Einstein himself seemed rather indifferent to the experimental tests, however. The first came in 1919, when the British physicist Arthur Eddington observed the Sun’s gravity bending starlight during a solar eclipse. What if those results hadn’t agreed with the theory? (Some accuse Eddington of cherry-picking the figures anyway, but that’s another story.) ‘Then,’ said Einstein, ‘I would have been sorry for the dear Lord, for the theory is correct.’

That was Einstein all over. As the Danish physicist Niels Bohr commented at the time, he was a little too fond of telling God what to do. But this wasn’t sheer arrogance, nor parental pride in his theory. The reason Einstein felt general relativity must be right is that it was too beautiful a theory to be wrong.

This sort of talk both delights today’s physicists and makes them a little nervous. After all, isn’t experiment – nature itself – supposed to determine truth in science? What does beauty have to do with it? ‘Aesthetic judgments do not arbitrate scientific discourse,’ the string theorist Brian Greene reassures his readers in The Elegant Universe (1999), the most prominent work of physics exposition in recent years. ‘Ultimately, theories are judged by how they fare when faced with cold, hard, experimental facts.’ Einstein, Greene insists, didn’t mean to imply otherwise – he was just saying that beauty in a theory is a good guide, an indication that you are on the right track.

Einstein isn’t around to argue, of course, but I think he would have done. It was Einstein, after all, who said that ‘the only physical theories that we are willing to accept are the beautiful ones’. And if he was simply defending theory against too hasty a deference to experiment, there would be plenty of reason to side with him – for who is to say that, in case of a discrepancy, it must be the theory and not the measurement that is in error? But that’s not really his point. Einstein seems to be asserting that beauty trumps experience come what may.

He wasn’t alone. Here’s the great German mathematician Hermann Weyl, who fled Nazi Germany to become a colleague of Einstein’s at the Institute of Advanced Studies in Princeton: ‘My work always tries to unite the true with the beautiful; but when I had to choose one or the other, I usually chose the beautiful.’ So much for John Keats’s ‘Beauty is truth, truth beauty.’ And so much, you might be tempted to conclude, for scientists’ devotion to truth: here were some of its greatest luminaries, pledging obedience to a different calling altogether.

Was this kind of talk perhaps just the spirit of the age, a product of fin de siècle romanticism? It would be nice to think so. In fact, the discourse about aesthetics in scientific ideas has never gone away. Even Lev Landau and Evgeny Lifshitz, in their seminal but pitilessly austere midcentury Course of Theoretical Physics, were prepared to call general relativity ‘probably the most beautiful of all existing theories’. Today, popularisers such as Greene are keen to make beauty a selling point of physics. Writing in this magazine last year, the quantum theorist Adrian Kent speculated that the very ugliness of certain modifications of quantum mechanics might count against their credibility. After all, he wrote, here was a field in which ‘elegance seems to be a surprisingly strong indicator of physical relevance’.

We have to ask: what is this beauty they keep talking about?

Some scientists are a little coy about that. The Nobel Prize-winning physicist Paul Dirac agreed with Einstein, saying in 1963 that ‘it is more important to have beauty in one’s equations than to have them fit experiment’ (how might Greene explain that away?). Yet faced with the question of what this all-important beauty is, Dirac threw up his hands. Mathematical beauty, he said, ‘cannot be defined any more than beauty in art can be defined’ – though he added that it was something ‘people who study mathematics usually have no difficulty in appreciating’. That sounds rather close to the ‘good taste’ of his contemporaneous art critics; we might fear that it amounts to the same mixture of prejudice and paternalism.

Given this history of evasion, it was refreshing last November to hear the theoretical physicist Nima Arkani-Hamed spell out what ‘beauty’ really means for him and his colleagues. He was talking to the novelist Ian McEwan at the Science Museum in London, during the opening of the museum’s exhibition on the Large Hadron Collider. ‘Ideas that we find beautiful,’ Arkani-Hamed explained, ‘are not a capricious aesthetic judgment’:

It’s not fashion, it’s not sociology. It’s not something that you might find beautiful today but won’t find beautiful 10 years from now. The things that we find beautiful today we suspect would be beautiful for all eternity. And the reason is, what we mean by beauty is really a shorthand for something else. The laws that we find describe nature somehow have a sense of inevitability about them. There are very few principles and there’s no possible other way they could work once you understand them deeply enough. So that’s what we mean when we say ideas are beautiful.

Does this bear any relation to what beauty means in the arts? Arkani-Hamed had a shot at that. Take Ludwig van Beethoven, he said, who strove to develop his Fifth Symphony in ‘perfect accordance to its internal logical structure’.

it is precisely this that delights mathematicians in a great proof: not that it is correct but that it shows a tangibly human genius

Beethoven is indeed renowned for the way he tried out endless variations and directions in his music, turning his manuscripts into inky thickets in his search for the ‘right’ path. Novelists and poets, too, can be obsessive in their pursuit of the mot juste. Reading the novels of Patrick White or the late works of Penelope Fitzgerald, you get the same feeling of almost logical necessity, word by perfect word.

But you notice this quality precisely because it is so rare. What generally brings a work of art alive is not its inevitability so much as the decisions that the artist made. We gasp not because the words, the notes, the brushstrokes are ‘right’, but because they are revelatory: they show us not a deterministic process but a sensitive mind making surprising and delightful choices. In fact, pure mathematicians often say that it is precisely this quality that delights them in a great proof: not that it is correct but that it shows a personal, tangibly human genius taking steps in a direction we’d never have guessed.

‘The things that we find beautiful today we suspect would be beautiful for all eternity’: here is where Arkani-Hamed really scuppers the notion that the kind of beauty sought by science has anything to do with the major currents of artistic culture. After all, if there’s one thing you can say about beauty, it is that the beholder has a lot to do with it. We can still find beauty in the Paleolithic paintings at Lascaux and the music of William Byrd, while admitting that a heck of a lot of beauty really is fashion and sociology. Why shouldn’t it be? How couldn’t it be? We still swoon at Jan van Eyck. Would van Eyck’s audience swoon at Mark Rothko?

The gravest offenders in this attempted redefinition of beauty are, of course, the physicists. This is partly because their field has always been heir to Platonism – the mystical conviction of an orderly cosmos. Such a belief is almost a precondition for doing physics in the first place: what’s the point in looking for rules unless you believe they exist? The MIT physicist Max Tegmark now goes so far as to say that mathematics constitutes the basic fabric of reality, a claim redolent of Plato’s most extreme assertions in Timaeus.

But Platonism will not connect you with the mainstream of aesthetic thought – not least because Plato himself was so distrustful of art (he banned the lying poets from his Republic, after all). Better that we turn to Immanuel Kant. Kant expended considerable energies in his Critique of Judgment (1790) trying to disentangle the aesthetic aspects of beauty from the satisfaction one feels in grasping an idea or recognising a form, and it does us little good to jumble them up again. All that conceptual understanding gives us, he concluded, is ‘the solution that satisfies the problem… not a free and indeterminately final entertainment of the mental powers with what is called beautiful’. Beauty, in other words, is not a resolution: it opens the imagination.

Physicists might be the furthest gone along Plato’s trail, but they are not alone. Consider the many chemists whose idea of beauty seems to be dictated primarily by the molecules they find pleasing – usually because of some inherent mathematical symmetry, such as in the football-shaped carbon molecule buckminsterfullerene (strictly speaking, a truncated icosahedron). Of course, this is just another instance of mathematics-worship, yoking beauty to qualities of regularity that were not deemed artistically beautiful even in antiquity. Brian Greene claims: ‘In physics, as in art, symmetry is a key part of aesthetics.’ Yet for Plato it was precisely art’s lack of symmetry (and thus intelligibility) that denied it access to real beauty. Art was just too messy to be beautiful.

In seeing matters the other way around, Kant speaks for the mainstream of artistic aesthetics: ‘All stiff regularity (such as approximates to mathematical regularity) has something in it repugnant to taste.’ We weary of it, as we do a nursery rhyme. Or as the art historian Ernst Gombrich put it in 1988, too much symmetry ensures that ‘once we have grasped the principle of order… it holds no more surprise’. Artistic beauty, Gombrich believed, relies on a tension between symmetry and asymmetry: ‘a struggle between two opponents of equal power, the formless chaos, on which we impose our ideas, and the all-too-formed monotony, which we brighten up by new accents’. Even Francis Bacon (the 17th-century proto-scientist, not the 20th-century artist) understood this much: ‘There is no excellent beauty that hath not some strangeness in the proportion.’

Perhaps I have been a little harsh on the chemists – those cube- and prism-shaped molecules are fun in their own way. But Bacon, Kant and Gombrich are surely right to question their aesthetic merit. As the philosopher of chemistry Joachim Schummer pointed out in 2003, it is simply parochial to redefine beauty as symmetry: doing so cuts one off from the dominant tradition in artistic theory. There’s a reason why our galleries are not, on the whole, filled with paintings of perfect spheres.

Why shouldn’t scientists be allowed their own definition of beauty? Perhaps they should. Yet isn’t there a narrowness to the standard that they have chosen? Even that might not be so bad, if their cult of ‘beauty’ didn’t seem to undermine the credibility of what they otherwise so strenuously assert: the sanctity of evidence. It doesn’t matter who you are, they say, how famous or erudite or well-published: if your theory doesn’t match up to nature, it’s history. But if that’s the name of the game, why on earth should some vague notion of beauty be brought into play as an additional arbiter?

Because of experience, they might reply: true theories are beautiful. Well, general relativity might have turned out OK, but plenty of others have not. Take the four-colour theorem: the proposal that it is possible to colour any arbitrary patchwork in just four colours without any patches of the same colour touching one another. In 1879 it seemed as though the British mathematician Alfred Kempe had found a proof – and it was widely accepted for a decade, because it was thought beautiful. It was wrong. The current proof is ugly as heck – it relies on a brute-force exhaustive computer search, which some mathematicians refuse to accept as a valid form of demonstration – but it might turn out to be all there is. The same goes for Andrew Wiles’s proof of Fermat’s Last Theorem, first announced in 1993. The basic theorem is wonderfully simple and elegant, the proof anything but: 100 pages long and more complex than the Pompidou Centre. There’s no sign of anything simpler.

It’s not hard to mine science history for theories and proofs that were beautiful and wrong, or complicated and right. No one has ever shown a correlation between beauty and ‘truth’. But it is worse than that, for sometimes ‘beauty’ in the sense that many scientists prefer – an elegant simplicity, to put it in crude terms – can act as a fake trump card that deflects inquiry. In one little corner of science that I can claim to know reasonably well, an explanation from 1959 for why water-repelling particles attract when immersed in water (that it’s an effect of entropy, there being more disordered water molecules when the particles stick together) was so neat and satisfying that it continues to be peddled today, even though the experimental data show that it is untenable and that the real explanation probably lies in a lot of devilish detail.

I would be thrilled if the artist were to say to the scientist: ‘No, we’re not even on the same page’

Might it even be that the marvellous simplicity and power of natural selection strikes some biologists as so beautiful an idea – an island of order in a field otherwise beset with caveats and contradictions – that it must be defended at any cost? Why else would attempts to expose its limitations, exceptions and compromises still ignite disputes pursued with near-religious fervour?

The idea that simplicity, as distinct from beauty, is a guide to truth – the idea, in other words, that Occam’s Razor is a useful tool – seems like something of a shibboleth in itself. As these examples show, it is not reliably correct. Perhaps it is a logical assumption, all else being equal. But it is rare in science that all else is equal. More often, some experiments support one theory and others another, with no yardstick of parsimony to act as referee.

We can be sure, however, that simplicity is not the ultimate desideratum of aesthetic merit. Indeed, in music and visual art, there appears to be an optimal level of complexity below which preference declines. A graph of enjoyment versus complexity has the shape of an inverted U: there is a general preference for, say, ‘Eleanor Rigby’ over both ‘Baa Baa Black Sheep’ and Pierre Boulez’s Structures Ia, just as there is for lush landscapes over monochromes. For most of us, our tastes eschew the extremes.

Ironically, the quest for a ‘final theory’ of nature’s deepest physical laws has meant that the inevitability and simplicity that Arkani-Hamed prizes so highly now look more remote than ever. For we are now forced to contemplate no fewer than 10500 permissible variants of string theory. It’s always possible that 10500 minus one of them might vanish at a stroke, thanks to the insight of some future genius. Right now, though, the dream of elegant fundamental laws lies in bewildering disarray.

An insistence that the ‘beautiful’ must be true all too easily elides into an empty circularity: what is true must therefore be beautiful. I see this in the conviction of many chemists that the periodic table, with all its backtracking sequences of electron shells, its positional ambiguities for elements such as hydrogen and unsightly bulges that the flat page can’t constrain, is a thing of loveliness. There, surely, speaks the voice of duty, not genuine feeling. The search for an ideal, perfect Platonic form of the table amid spirals, hypercubes and pyramids has an air of desperation.

Despite all this, I don’t want scientists to abandon their talk of beauty. Anything that inspires scientific thinking is valuable, and if a quest for beauty – a notion of beauty peculiar to science, removed from art – does that, then bring it on. And if it gives them a language in which to converse with artists, rather than standing on soapboxes and trading magisterial insults like C P Snow and F R Leavis, all the better. I just wish they could be a bit more upfront about the fact that they are (as is their wont) torturing a poor, fuzzy, everyday word to make it fit their own requirements. I would be rather thrilled if the artist, rather than accepting this unified pursuit of beauty (as Ian McEwan did), were to say instead: ‘No, we’re not even on the same page. This beauty of yours means nothing to me.’

If, on the other hand, we want beauty in science to make contact with aesthetics in art, I believe we should seek it precisely in the human aspect: in ingenious experimental design, elegance of theoretical logic, gentle clarity of exposition, imaginative leaps of reasoning. These things are not vital for a theory that works, an experiment that succeeds, an explanation that enchants and enlightens. But they are rather lovely. Beauty, unlike truth or nature, is something we make ourselves.