The Buddhist monks from Namgyal monastery in India engage in a ritual that involves the creation of intricate patterns of coloured sand, known as mandalas. As large as three metres across, each mandala requires a couple of weeks of painstaking work, in which several monks in orange robes bend over a flat surface and scratch metallic vials. The vials extrude sand from tiny spouts, a few grains at a time, onto areas bounded by carefully measured chalk marks. Slowly, slowly, the ancient pattern is made. After the thing is completed, the monks say a prayer, pause a moment, and then sweep it all up in five minutes.

Although I haven’t witnessed this particular ritual, I’ve seen a number of mandalas during my travels in Southeast Asia. For Buddhists, the creation and destruction of a mandala symbolises the impermanence of earthly existence. But the ritual also reminds me of the profound symbiosis of order and disorder at the core of our world.

Somewhat surprisingly, nature not only requires disorder but thrives on it. Planets, stars, life, even the direction of time all depend on disorder. And we human beings as well. Especially if, along with disorder, we group together such concepts as randomness, novelty, spontaneity, free will and unpredictability. We might put all of these ideas in the same psychic basket. Within the oppositional category of order, we can gather together notions such as systems, law, reason, rationality, pattern, predictability. While the different clusters of concepts are not mirror images of one another, like twilight and dawn, they have much in common.

Our primeval attraction to both order and disorder shows up in modern aesthetics. We like symmetry and pattern, but we also relish a bit of asymmetry. The British art historian Ernst Gombrich believed that, although human beings have a deep psychological attraction to order, perfect order in art is uninteresting. ‘However we analyse the difference between the regular and the irregular,’ he wrote in The Sense of Order (1979), ‘we must ultimately be able to account for the most basic fact of aesthetic experience, the fact that delight lies somewhere between boredom and confusion.’ Too much order, we lose interest. Too much disorder, and there’s nothing to be interested in. My wife, a painter, always puts a splash of colour in the corner of her canvas, off balance, to make the painting more appealing. Evidently, our visual sweet-spot lies somewhere between boredom and confusion, predictability and newness.

Human beings have a conflicted relationship to this order-disorder nexus. We are alternately attracted from one to the other. We admire principles and laws and order. We embrace reasons and causes. We seek predictability. Some of the time. On other occasions, we value spontaneity, unpredictability, novelty, unconstrained personal freedom. We love the structure of Western classical music, as well as the free-wheeling runs or improvised rhythms of jazz. We are drawn to the symmetry of a snowflake, but we also revel in the amorphous shape of a high-riding cloud. We appreciate the regular features of pure-bred animals, while we’re also fascinated by hybrids and mongrels. We might respect those who manage to live sensibly and lead upright lives. But we also esteem the mavericks who break the mould, and we celebrate the wild, the unbridled and the unpredictable in ourselves. We are a strange and contradictory animal, we human beings. And we inhabit a cosmos equally strange.

You can see the creative tension of the order-disorder nexus in our science versus our art. In his law of floating bodies, formulated in 250 BCE, Archimedes prefigured the coming age of science when he expressed one of the first quantitative laws of nature: ‘Any body wholly or partially immersed in a fluid experiences an upward force equal to the weight of the fluid displaced.’ In other words, a body sinks just to the level where the weight of the displaced fluid equals the weight of the body. To verify this elegant law, Archimedes would have done the experiment over and over with various objects of different shapes and sizes, and with different liquids such as water and mercury. (Scales were available in the Greek agora for weighing wheat, salted fish, glass, copper and silver.)

Evidently, the world of masses and forces was logical, rational, quantifiable, predictable. Yet two centuries earlier, Socrates – that wandering sage that Plato and others described as resembling a satyr more than a man, short and stocky, with a pug nose and bulging eyes – celebrates the creative power of madness: ‘He who, having no touch of the Muses’ madness in his soul, comes to the door and thinks that he will get into the temple by the help of art – he, I say, and his poetry are not admitted; the sane man disappears and is nowhere when he enters into rivalry with the madman.’

Creativity has always been associated with novelty, surprise, and what psychologists and neuroscientists call divergent thinking: the ability to explore many different avenues and solutions to a problem in a spontaneous and non-orderly fashion. Convergent thinking, by contrast, is the more logical and orderly step-by-step approach to a problem. The French mathematician Henri Poincaré in 1910 described the gestation of one of his mathematical discoveries as a dance between the two:

For 15 days, I strove to prove that there could not be any [mathematical] functions like those I have since called Fuchsian functions. I was then very ignorant; every day I seated myself at my work table, stayed an hour or two, tried a great number of combinations and reached no results. One evening, contrary to my custom, I drank black coffee and could not sleep. Ideas rose in crowds; I felt them collide until pairs interlocked, so to speak, making a stable combination. By the next morning…

Undoubtedly, some of our creativity is ignited by a synthesis of convergence and divergence, working together in symphony.

The critical role of disorder in nature was not articulated until 2,000 years after Socrates praised the mad poet. The task fell to the German physicist Rudolf Clausius. He was born in 1822 in Pomerania, a region split between Germany and Poland, and educated at the University of Berlin. Perhaps under the influence of his religious father, a reverend, Clausius led a principled life. ‘A chief characteristic was his sincerity and fidelity,’ was how his brother, Robert, described Clausius at his death in 1888. ‘Every kind of exaggeration was opposed to his nature.’

Like Albert Einstein, Clausius was a theoretical physicist – that is, all of his work, including his seminal work on disorder, consisted of mathematical feats performed with pencil and paper. Clausius’s great paper on disorder, ‘On the Moving Force of Heat’ (1850), was published the same year that he became a professor of physics at the Royal Artillery and Engineering School in Berlin. In that paper, Clausius showed that change in the physical world is associated with the inevitable movement of order to disorder. Indeed, without the potential of disorder, nothing in the cosmos would ever change – like a row of upright dominoes held rigidly in place, or like a completed Buddhist mandala locked in a bank vault, safe from the brooms of the monks of Namgyal.

‘Heat’ occurs in the title of Clausius’s paper because increasing disorder is often associated with the transfer of heat from hot bodies to cold – but the concept is more general. In a later paper, Clausius coined the term entropy as a quantitative measure of disorder. The word comes from the Greek ἐν (en), meaning ‘in’, and τροπή (tropē), meaning ‘transformation’. It is the increase of entropy that is linked to transformation, movement, change in the world. The more disorder, the more entropy. The last two sentences of Clausius’s 1850 paper are:

1. The energy of the Universe is constant

2. The entropy of the Universe tends toward a maximum

Order inevitably yields to disorder, and entropy increases until it cannot increase any further. It is this movement that drives the world. Clean rooms become dusty. Temples slowly crumble. As we grow older, bones grow brittle. Stars eventually burn out, emptying their hot energy into the coldness of space – but while doing so, they provide warmth and life to surrounding planets. We live off this relentless increase of disorder.

Disorder is also the answer to the profound question: why is there something rather than nothing?

Even something as fundamental as the direction of time is governed by the movement of order to disorder. If that seems like a preposterous statement, consider a glass goblet falling off a table and shattering on the floor – a transformation from order to disorder of the most obvious kind. A film of the event would look normal to us. But if we saw a movie of scattered shards of glass jumping off the floor and gathering themselves into a fully formed goblet, perched on the edge of the table, we would say the movie was being played backwards in time. Why? Because everything passes from order to disorder as we march towards the future. One might say that the forward direction of time is the increase in disorder. Indeed, without these changes, we’d have no way of telling one instant from the next. There would be no clocks, no flights of birds, no leaves slipping through the air as they dropped from trees, no breathing out and in. The Universe would be a still photo for all of eternity.

Disorder is also the answer to the profound question: why is there something rather than nothing? (Such questions keep physicists and philosophers up at night.) Why does material of any kind exist, rather than pure energy? From a scientific perspective, the question relates to the existence of antiparticles, predicted in 1931 and then discovered in 1932. Every subatomic particle, such as the electron, has an antiparticle twin – identical to the first, except with opposite electrical charge and certain other qualities. Which of the pair we call the ‘particle’ and which we call the ‘antiparticle’ is a matter of convention, like the North and South poles. When they meet, particles and their antiparticles annihilate each other, leaving nothing but pure energy.

If there were an equal number of particles and their antiparticles in the infant Universe, as one would expect from a completely symmetrical universe, all matter would have been obliterated billions of years ago, leaving nothing but pure energy. No stars, no planets, no people – or any other solid material. So why are we here? Why haven’t all the particles disappeared along with their antiparticle partners?

The answer to this physicists’ conundrum came in 1964. In very delicate experiments at that time, we discovered that particles and antiparticles do not behave in exactly the same way. Rather, there is a slight asymmetry in how they interact with other particles, so that immediately after the creation of the Universe, particles and their antiparticles were not produced and destroyed in equal numbers. After the mass annihilations of particles with their antiparticle partners, some particles would remain, like a surplus of boys sitting on the bench at a school dance. Those remaining particles and the asymmetry that produced them is why we exist.

Disorder isn’t just present in the minutiae of how matter organises itself. It also runs deep within the structures of life itself. Perhaps the most well-known example of disorder in biology is the shuffling of genes – both by mutations, and by the transfer of genes from viruses and other organisms. Through these random processes, living organisms try out different bodily architectures that might have never been sampled otherwise. These spins of the genetic roulette wheel aren’t planned, and their outcomes can’t be known in advance. But without them, biology would be stuck with a small number of inflexible designs. Many organisms would die out, unable to adapt to changing environmental conditions, and there would be much less biodiversity on Earth.

Another significant way that disorder makes itself known in biology occurs via a process called diffusion. Here, a lumpy gobbet of matter or energy is automatically smoothed out by the random collisions of atoms and molecules. You can see this for yourself if you pour a bucket of hot water into a cool bath. At first, the bath will have a hot region surrounded by a cool region. But the hot water will quickly mix with the cool until the bath comes to a uniform temperature. That’s diffusion. To paraphrase Clausius, diffusion doesn’t cost any energy, but it increases disorder – in this case, mixing heat – which drives transformation and change. Without random molecular collisions, diffusion would not occur. The hot water would remain at one side of the tub, and the cool water on the other.

Diffusion is a key mechanism for transporting vital substances throughout the body. Take oxygen, the essential gas for energy production. With each inhalation, we produce a high concentration of oxygen in our lungs. The tiny blood vessels embedded in the lungs have a relatively low amount of oxygen. That allows the vital gas to ‘diffuse’ from the lungs to the blood, and then, for the same reason, from the blood to individual cells throughout the body. Such directed movement results from the random collisions, tending to transport oxygen molecules from areas of high to low oxygen concentration. Without random bumps and knocks, going this way and that, oxygen in the lungs would remain trapped in the lungs, and the cells of the body would suffocate.

The signals that race between our nerve cells are another biological example of diffusion. When electrically charged sodium and potassium atoms move across the outer wall of nerve cells, they create a pulse of electricity. Such movement, in turn, is caused by the random shuffling of a high concentration of charged atoms to a region of lower concentration, evening out the concentration of the atoms. Ironically enough, the random collisions of individual atoms are what lead to the orderly progression of a nerve impulse down the nerve. This is the mechanism by which the body communicates with itself.

But none of these examples from the microscopic domain, including Clausius’s deep pronouncements on entropy, explain humans’ paradoxical attraction to both order and disorder – our honouring of both the respectable and the mavericks among us. There seems to be something deep in our psyche, something primeval, imprinted in us aeons before Clausius or Socrates. Perhaps the embrace of these opposites conferred an adaptive advantage on our ancestors, many millions of years in our past.

The conjecture seems sound. From an evolutionary point of view, order implies predictability, patterns, repeatability – all of which allow us to make good predictions. And predictions are useful for knowing when game will run through the forest, or when crops should be planted. The benefit to our survival is obvious. More unexpected, perhaps, is how attentiveness to surprise, chance and novelty can also confer an advantage. If we get too complacent with our routine, we can’t react when things change, when the tiger suddenly appears on the path that we have walked along a thousand times without mishap. And we would not take risks, for fear of departing from our familiar routines. So it makes sense that we’ve developed a desire for both the predictable and the unpredictable.

Since order and disorder benefit human beings, it’s worth re-examining why we divide everything into polar opposites

If an appetite for novelty conferred a survival benefit on our ancestors, perhaps it should show up in our genes. Researchers have recently discovered a variation (allele) of a gene called DRD4-7R – or more arrestingly, ‘the wanderlust gene’. It occurs in about 20 per cent of the population, and appears to be associated with a penchant for exploration and risk. It makes sense that we’d want most members of our tribe to stay at home, follow routine, tend to the hearth. Yet we also need a few others to venture forth on risky expeditions in search of new hunting grounds and unexpected opportunities. ‘We have evidence to suggest that the same allele involved in the personality trait of novelty-seeking and impulsivity was also involved in being pro-risk in financial situations,’ says Richard Paul Ebstein, professor of psychology at the National University of Singapore and one of the leading researchers of DRD4-7R. ‘People who have that allele appear to be more risk-prone.’ Other biologists rightly point out that it’s unlikely that any single gene could control a trait such as risk-taking and novelty-seeking – but a group of genes working together might just do so.

Since both order and disorder evidently benefit human beings, it’s worth re-examining our inclination, at least in the West, to divide everything into polar opposites, with an assumed hierarchy of value and unstated preferences – productivity and laziness, rationality and irrationality, hot and cold, smooth and rough, white and black. Perhaps, instead, we should view such opposites in terms of a useful balance.

The Danish physicist Niels Bohr, one of the pioneers of quantum physics, once said that the opposite of a profound truth is also true. The Chinese have long understood this idea in terms of the ancient Daoist and Confucian concept of yin and yang: all things exist as inseparable and contradictory opposites. Yin is associated with feminine, dark, north, old, soft, cold, while yang with masculine, light, south, young, hard, warm. The symbol of yin and yang – two entangled swirls, one black and one white, equal in size, each with a dot of the other colour within it – suggests that the two exist in harmony, with neither dominant over the other. Meanwhile, Western thought typically attempts to simplify this baffling world by dividing everything into two. That works for a while, until we look more closely and discover the real complexity lurking underneath. If eventually we are able to stand on higher ground, we once again find simplicity and harmony. The cosmos sings order, and it also sings disorder. We human beings seek predictability, and we also yearn for the new. Embrace these necessary contradictions, say Bohr and the Confucians.

It is the end of my writing day, and I am listening to Anton Bruckner’s Ninth Symphony, which the Austrian composer began writing in 1887. The symphony opens with a continuous unfolding of the themes. The second movement, the Scherzo, feels sinister, as if some dark secret is being withheld. But I find myself mesmerised by a section of the third movement, the Adagio. After a haunting and harmonious melody from the strings (perhaps promising the coming secret), the sounds become increasingly discordant, building in volume, until we hear a thunderclap of the horns, jagged and dissonant, followed by more clashes, like tidal waves pounding the shore. Then a moment of silence. The strings pick up again, quiet and lyrical. This alternation of the melodious with the dissonant continues until the end of the movement. And I wonder if the harmonious sections of the piece would be quite so beautiful if not juxtaposed with the unharmonious, the light with the dark, the smooth with the rough. The orderly with the seemingly disorderly. And of course Bruckner himself – a chance event like all of us, a random collision of cells bringing forth improbable life in this improbable Universe.