I stop and do nothing. Nothing happens. I am thinking about nothing. I listen to the passing of time. This is time, familiar and intimate. We are taken by it. The rush of seconds, hours, years that hurls us towards life then drags us towards nothingness ... We inhabit time as fish live in water. Our being is being in time. Its solemn music nurtures us, opens the world to us, troubles us, frightens and lulls us. The universe unfolds into the future, dragged by time, and exists according to the order of time. What could be more universal and obvious than this flowing?

Benedict Cumberbatch reads The Order of Time.

Reality is often very different from what it seems. The Earth appears to be flat but is in fact spherical. The sun seems to revolve in the sky when it is really we who are spinning. Neither is time what it seems to be.

Let’s begin with a simple fact: time passes faster in the mountains than it does at sea level. The difference is small but can be measured with precision timepieces that can be bought today for a few thousand pounds. This slowing down can be detected between levels just a few centimetres apart: a clock placed on the floor runs a little more slowly than one on a table.

It is not just the clocks that slow down: lower down, all processes are slower. Two friends separate, with one of them living in the plains and the other going to live in the mountains. They meet up again years later: the one who has stayed down has lived less, aged less, the mechanism of his cuckoo clock has oscillated fewer times. He has had less time to do things, his plants have grown less, his thoughts have had less time to unfold ... Lower down, there is simply less time than at altitude.

Carlo Rovelli. Photograph: Roberto Serra / Iguana Press / G/Iguana Press / Getty Images

Einstein understood this slowing down of time a century before we had clocks precise enough to measure it. He imagined that the sun and the Earth each modified the space and time that surrounded them, just as a body immersed in water displaces the water around it. This modification of the structure of time influences in turn the movement of bodies, causing them to “fall” towards each other.

What does it mean, this “modification of the structure of time”? It means precisely the slowing down of time described above: a mass slows down time around itself. The Earth is a large mass and slows down time in its vicinity. It does so more in the plains and less in the mountains, because the plains are closer to it. This is why the friend who stays at sea level ages more slowly.

If things fall, it is due to this slowing down of time. Where time passes uniformly, in interplanetary space, things do not fall. They float. Here on the surface of our planet, on the other hand, the movement of things inclines naturally towards where time passes more slowly, as when we run down the beach into the sea and the resistance of the water on our legs makes us fall headfirst into the waves. Things fall downwards because, down there, time is slowed by the Earth.

Times are legion: a different one for every point in space

In a physics laboratory, a clock on a table and another on the ground run at different speeds. Which tells the time? The question is meaningless. We might just as well ask what is most real – the value of sterling in dollars or the value of dollars in sterling. There are two times that change relative to each other. Neither is truer than the other. But there are not just two times. Times are legion: a different one for every point in space. The single quantity “time” melts into a spiderweb of times. We do not describe how the world evolves in time: we describe how things evolve in local time, and how local times evolve relative to each other.

It only takes a few micrograms of LSD to expand our experience of time to an epic and magical scale. “How long is forever?” asks Alice. “Sometimes, just one second,” replies the White Rabbit. There are dreams lasting an instant in which everything seems frozen for an eternity. Time is elastic in our personal experience of it. Hours fly by like minutes, and minutes are oppressively slow, as if they were centuries.

Before Einstein told us that it wasn’t true, how the devil did we get it into our heads that time passes everywhere at the same speed? It was certainly not our direct experience of the passage of time that gave us the idea that time elapses at the same rate, always and everywhere.

Reality check: the White Rabbit consults his watch, in Alice in Wonderland. Photograph: Print Collector/Getty Images

So where did we get it from? For centuries, we have divided time into days. The word “time” derives from an Indo-European root – di or dai – meaning “to divide”. For centuries, we have divided the days into hours. For most of those centuries, however, hours were longer in the summer and shorter in the winter, because the 12 hours divided the time between dawn and sunset: the first hour was dawn, and the 12th was sunset, regardless of the season.

Sundials, hourglasses and water clocks already existed in the ancient world, in the Mediterranean region and in China – but they did not play the cruel role that clocks have today in the organisation of our lives. It is only in the 14th century in Europe that people’s lives start to be regulated by mechanical clocks. Cities and villages build their churches, and place a clock on the belltower to mark the rhythm of collective activities. The era of clock-regulated time begins. Gradually, time slips from the hands of the angels and into those of the mathematicians – as is graphically illustrated by Strasbourg Cathedral, where two sundials are surmounted, respectively, by an angel (one inspired by earlier sundials from 1200) and by a mathematician (on the sundial put there in 1400).

The usefulness of clocks supposedly resides in the fact that they tell the same time. And yet this idea is also more modern than we might imagine. For centuries, as long as travel was on horseback, on foot or in carriages, there was no reason to synchronise clocks between one place and another. There was good reason for not doing so. Midday is, by definition, when the sun is at its highest. Every city and village had a sundial that registered the moment the sun was at its midpoint, allowing the clock on the bell-tower to be regulated with it, for all to see. But the sun does not reach midday at the same moment in Venice, or in Florence, or in Turin, because the sun moves from east to west. Midday arrives first in Venice, and significantly later in Turin, and for centuries the clocks in Venice were a good half hour ahead of those in Turin. Every small village had its own peculiar “hour”. The station in Paris kept its own hour, a little behind the rest of the city, as a kind of courtesy towards travellers who were running late.

In the 19th century the telegraph arrives, trains become commonplace and fast, and the problem arises of properly synchronising clocks between one city and another. It is awkward to organise train timetables if each station marks time differently.

One of the sun dials on Strasbourg Cathedral. Photograph: Handout

It is in the United States that the first attempt is made to standardise time. Initially, it is proposed to fix a universal hour for the entire world. To call, for instance, “12 o’clock” the moment at which it is midday in London, so that midday would fall at 12.00 in London and around 18.00 in New York. The proposal is not well received, because people are attached to local time. In 1883 a compromise is reached with the idea of dividing the world into time zones, thereby standardising time only within each zone. In this way, the discrepancy between 12 on the clock and local midday is limited to a maximum of about 30 minutes. The proposal is gradually accepted by the rest of the world and clocks begin to be synchronised between different cities.

It can hardly be pure coincidence that, before gaining a university position, the young Einstein worked in the Swiss Patent Office, dealing specifically with patents relating to the synchronisation of clocks at railway stations. It was probably there that it dawned on him: the problem of synchronising clocks was, ultimately, an insoluble one.

In other words, only a few years passed between the moment at which we agreed to synchronise clocks and the moment at which Einstein realised that it was impossible to do so exactly.

• The Order of Time is published by Allen Lane. To order a copy for £9.75 (RRP £12.99) go to guardianbookshop.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min p&p of £1.99.