Alan Guth still finds it amazing that he can understand anything about the first moments of the Big Bang. But he shouldn’t – he was there.

About 13.8 billion years ago, when the Universe was a hundredth of a billionth of a trillionth of a trillionth of a second old, it experienced an extraordinary growth spurt, doubling in size more than 60 times over in a split second. This cosmic fireball quickly slowed, and after about 380,000 years cooled enough for electrons to combine with nuclei and form atoms. This liberated photons of light. Suddenly the Universe could be seen.

Somewhere between 150 million and one billion years later, gravity drove enough clumps of gas to collapse inward so that they formed the first stars. The intense heat and pressure deep within the stars acted like thermonuclear furnaces, converting the only matter that existed – hydrogen, helium and lithium – into heavier elements such as carbon, iron and nickel. And Alan Guth.

Not the astrophysicist himself, who sits before me in a checked blue shirt, light green chinos and tussled grey hair with a casual, toothy half-grin. But all the oxygen, carbon, hydrogen, nitrogen, calcium, phosphorus and trace elements that make up the man. The protons, neutrons and electrons of which those elements consist – themselves composed of subatomic fermions such as quarks and leptons – were made in that early universe.

“I find it absolutely amazing,” Guth, a professor of physics at the Massachusetts Institute of Technology, tells me. “I mean, we’re making theories about what was happening in the Universe at 10-38 seconds, which is totally off-scale, way beyond our experience. And nonetheless these predictions turned out to describe the fluctuations of the cosmic background radiation to incredible precision.”

For a giant in his field, who in September received the $1million Kavli Prize for Astrophysics, the 67-year-old is subdued and unassuming. His shaggy haircut dates from the 1970s, and so does the theory that made his reputation. Today, our Universe is remarkably even. Guth’s explanation? A titanic growth spurt in the split second after its birth that inflated it like a balloon, leaving it smooth and even. He dubbed his theory “cosmic inflation”.

And yet, this is a man who considers his PhD a failure. When he made his astounding discovery he’d been stuck in the lower echelons as an untenured “post-doc” for years. “I would say my future was somewhat uncertain at that point,” he says, adjusting his gold-rimmed glasses.

Professor Alan Guth draws the story of creation in chalk graffiti on a blackboard at MIT in 2002. – Photograph ©Larry Fink

Because he was working on his doctorate at MIT, Guth avoided being drafted into the Vietnam War. His thesis was delivered in 1972, the same year conscription ended. It described how quarks combine to form elementary particles and was built on the then-popular belief that subatomic quarks were extremely heavy. Unfortunately, the new theory of quantum chromodynamics emerged shortly after, rendering the idea of heavy quarks and Guth’s PhD thesis, obsolete.

The Universe was too perfect. This was the problem plaguing physicists in the 1970s.

He married his high school sweetheart, Susan Tisch, and had a hard time landing a permanent job. Over the next nine years he took post-doctoral research positions at Princeton University, then Columbia, then Cornell, eventually ending up at Stanford University’s SLAC National Accelerator Laboratory near San Francisco. Over that time his work ranged widely from describing the maths of subatomic particles to the Big Bang. But it was at SLAC that all of this came together in his own personal big bang. He modestly describes hitting on the idea of cosmic inflation as “being at the right place at the right time”.

The Universe was too perfect. This was the problem plaguing physicists in the 1970s. The Big Bang was a compelling description of how it began, but for this to lead to what we see today the density of matter and energy needed to be a very precise value – to an accuracy of 15 decimal places – or else the Universe would either blast itself apart or collapse on itself. This was known as the “flatness problem”. Another difficulty was the “horizon problem”: the two edges of the observable universe are almost 28 billion light-years apart yet across that distance the temperature is remarkably uniform to within 0.007%. Since nothing can move faster than light, heat radiation could not have travelled between these horizons to even out the difference. Physicists were stumped.

Neither of these problems were on the mind of Henry Tye, a fellow postdoctoral physicist at Cornell, who convinced Guth to work on equations that might predict the number of magnetic monopoles in the early Universe. Magnets have two poles but exotic beasts with a single pole are theoretically possible according to the equations of Scottish physicist James Clerk Maxwell. Monopoles have yet to be seen in the real Universe, but if the early Universe was superhot, many would likely have been produced. Guth and Tye set about calculating how many. Their answer was surprising: even today, the Universe should be littered with magnetic monopoles. What’s more, the monopoles would be so gargantuan that the Universe would have slowed down extremely fast after the Big Bang.

And if their calculations were right, the Universe would be a mere 10,000 years old. That told them they were on the wrong path, because the Universe cannot be younger than the Earth! “So that was clearly an impossible prediction,” Guth recalls. He and Tye began to search for explanations that avoided the overproduction of monopoles. What if the Universe cooled extremely quickly after the Big Bang, reducing the emergence of so many monopoles, they wondered? Working in the study of his rented house one night in late 1979, Guth set about doing the numbers, and found that this did indeed skirt the monopoles and age of the Universe problem.

At the Kavli 2014 awards. Recipients, left to right, Alan Guth, Andrei Linde and Alexei Starobinsky. – 2014 Kavli Prize

But at two in the morning, he had a flash of insight. Solving the monopoles problem required the Universe to expand exponentially. “And I realised that this would solve the flatness problem,” he recalls. In his notebook that night he wrote “spectacular realisation”, with two bold rectangles around it.

It was the birth of the theory of cosmic inflation. The next morning Guth cycled to SLAC in record time. “I was very worried that there’d be some gigantic flaw, so I was very anxious to bounce the idea off colleagues to see if people could poke holes in it.” However, his idea held up. In fact, from discussions with physicists in the SLAC cafeteria over the following weeks, it became obvious that inflation not only had legs, but also nicely solved the horizon problem. “It was all packed together cheek-by-jowl at subatomic distances before expanding wildly, so inflation neatly bridged the world of the very small and the world of the unimaginably large, tying them together. So I became all the more excited,” he says. Guth was suddenly in demand.

From January 1980 he began speaking about his idea at universities and research institutes, inviting comment and criticism from the audience. He had always been slow to write a scientific paper, but in this case he had good cause – while he could explain how inflation began, he couldn’t make it end in a way that allowed stars and galaxies to form. Plus, his postdoc was coming to an end and he needed to find a job again – although this time the offers began coming to him. In April, after a day of job interviews, he had dinner at a Chinese restaurant. His fortune cookie read: “An exciting oppportunity awaits you if you are not too timid”.

Guth wanted to go back at MIT but there were no jobs on offer. He contacted a friend there, letting him know he was entertaining a host of job offers but would prefer to work in Boston. A day later, Guth had his job offer. “My wife was very happy,” he smiles. By June he’d finally completed his calculations on how inflation ended, but he wasn’t pleased – it predicted a lumpy universe, which is not what astronomers were seeing. He decided to publish his paper in January 1981, arguing that it was a powerful explanation despite the unsatisfying ending, and urged others to find ways to make the idea of cosmic inflation work.

President Obama greets the 2014 Kavli Prize winners in the Oval Office. Andrei Linde is to the left of the president and Alan Guth fourth from the left. – The White House

There was already a lot of excitement – reports of his talks on inflation had been circulating in the physics community – but no-one had a ready answer. Several attempts were made, but it actually took years for Guth’s challenge to be answered. It was Russian physicist Andrei Linde, then at the Lebedev Physical Institute in Moscow, who in 1983 proposed a revamped approach in which the Universe gracefully exits from the exponential expansion without producing a wildly clumpy structure. It electrified physicists and soon became the prototype of modern inflationary models. It also added a dollop of weirdness suggesting that our Universe is one of many inflationary universes that have sprouted into being – that there is a multiverse of all possible types existing, and still being created.

In cracking the problem, Linde was inspired by the work of fellow Russian Alexei Starobinsky of the Landau Institute of Theoretical Physics near Moscow, who in 1980 independently postulated exponential expansion, although driven by a different mechanism – quantum gravity effects. Published in Russian, Starobinsky’s paper was unknown to Guth and others in the West and did not tackle the flatness and horizon problems. Nevertheless, it was pivotal. Today inflation theory has been validated many times over by the WMAP (Wilkinson Microwave Anisotropy Probe) spacecraft and other experiments that map the cosmic microwave background – the ancient light released by the Big Bang that still glimmers through the Universe. The strength of the theory was recognised this September when all three men were awarded the Kavli Prize in Astrophysics by Norway’s king at a ceremony in Oslo.

That was where I caught up with Guth and Linde. Over coffee at the Grand Hotel in Oslo, the same place where playwright Henrik Ibsen used to eat every day, Guth gushed about how Linde closed the loop, “My version of inflation did not work. Andrei made it work.” Now a professor at Stanford, Linde is an ebullient man with windswept white hair and a wide grin, who credits Guth with “a tremendous change in perspective” in physics. “Before the theory of inflation, everyone thought that quantum mechanics had an effect at very small scales, but at large scales it was considered not to be relevant,” he told me. “But we learned that the largest objects in the Universe – galaxies – were produced by quantum fluctuations.” Not so long ago, “this would have sounded like a crazy idea, one that’s good for science fiction books but not for physics. Yet you go and measure, and everything just fits into this science fiction picture. It’s just amazing.” Starobinksy, a greying and bespectacled man with bushy eyebrows and a more reserved bearing, whom I also met after the Kavli ceremony, agrees. “With this theory, cosmology becomes more like other areas of physics, it becomes predictive – we can predict what we see today with great accuracy. That’s very exciting.”

All three admit to still being astounded that humans can understand anything about what the Universe was like when it was, as Starobinsky put it, “unimaginably small, less than 0.0000000000000000000000000001 centimetres across. That’s that’s 27 zeros between the decimal point and the one!”

In March this year a team led by Harvard astrophysicist John Kovac announced it had discovered gravitational waves using an Antarctica-based telescope, BICEP2 (for Background Imaging of Cosmic Extragalactic Polarisation 2). The fingerprint of gravitational waves was detectable as swirling patterns, dubbed B modes, in polarised light from the cosmic microwave background. The news rocked the physics world and greatly excited Guth, Linde and Starobinsky. Their inflation theory predicted gravity waves: space-time would have “bounced” in response to the shock, producing gravity waves which left their imprint as swirls in the cosmic microwave background. Gravitational waves were predicted by Albert Einstein’s 1916 Theory of General Relativity but are exceptionally feeble and difficult to detect. Despite decades of searching, no conclusive evidence of them had been discovered.

Shortly after the BICEP2 announcement doubts emerged. Critics suggested the tiny swirls might have nothing to do with gravity waves: they could have been caused by dust particles looping around the Milky Way’s magnetic field. They too can polarise light and create similar patterns. Kovac’s team countered that they used six different models to predict what dust patterns might look like, then subtracted them one at a time from the raw data. Theoretically, the signals left over would be those of bona fide gravitational waves. In September a new paper based on data collected by the European Space Agency’s Planck space observatory found much more dust than the BICEP2 team had anticipated in their models. In fact, enough to account for the swirls Kovac’s team measured – although the authors of the Planck paper stressed their analysis did not rule out BICEP2’s gravitational waves.

The news was disheartening. Measuring the primordial gravitational waves would have provided further validation of inflation theory, as well as offering hard numbers to explain how the Universe operates from the cosmic to the quantum scale. “Detection of gravity waves could point to the unification of several forces of nature,” says cosmologist Lawrence Krauss of Arizona State University. Guth is unperturbed that the BICEP2 data might be a mirage. What ruffles him is journalists dubbing the putative discovery as the only real proof for cosmic inflation. “When people said that gravitational waves would be the smoking gun for inflation, my response was that I thought the room was pretty filled with smoke already.”

But then Guth is not one to worry. For a man whose academic journey began with a failed PhD, he nevertheless struck upon a solution that made sense of the mess cosmology was in. Which is ironic, really, considering the legendary mess in his office. Among the many awards he’s picked up over the years is one given by The Boston Globe for the messiest office in town. He was nominated by colleagues who hoped it would shame him into tidying up. “It’s still pretty messy,” he admitted. Perhaps he needs the chaos.