Through his study of ancient DNA, the geneticist Svante Pääbo has identified the Neanderthal lurking in many of us – and turned paleontology on its head

Leipzig's Max Planck Institute of Evolutionary Anthropology is a striking edifice. Set among the drab housing blocks of former East Germany, the building, erected in 2003 and made largely of glass, curves gently like a banana round the edge of a suburban crescent. There is a pond in its huge forecourt with turtles and ducks. Television screens in its ground-floor cafeteria relay live images of orangutans and chimps from Leipzig zoo. There is a grand piano in one corner and above it a climbing wall rises to the main atrium's ceiling several floors overhead. For good measure, there is a sauna on the roof.

It is not clear whether staff relax by climbing the institute's walls to the sound of a Liszt concerto before taking a rooftop sauna. If they do, they have Svante Pääbo to thank. "The climbing wall and sauna were my ideas," he says. "When they put together plans for the institute and asked me work here, I insisted they have these. I suppose they fit in with my Swedish identity."

Pääbo is the institute's head of evolutionary genetics and an idiosyncratic researcher, a man obsessed – he admits – with the avoidance of contamination and the need to keep his laboratories and researchers scrupulously clean. In doing so, Pääbo has transformed the study of human origins. Among his achievements, he has sequenced an entire Neanderthal genome, revealing a link between these extinct people and many modern humans. He has also uncovered the existence of a previously unknown human species, called the Denisovans, from DNA extracted from a finger bone found in a cave in Siberia.

"When I started this field 25 years ago I thought we might be able to extract DNA from bones of people born a few thousand years ago and learn something about the ancient Egyptians or about the people who brought agriculture into Europe," says Pääbo. "It was beyond my wildest dreams to think we could resurrect genomes that are hundreds of thousands of years old. To have done that, well, it's really cool."

It is also incredibly difficult. DNA, the stuff from which our genes are made, decays the moment an organism dies. The long coils break down into fragments and the longer the passage of time, the shorter the fragments become. Trying to put these tiny pieces together is a stunningly complex task that has been likened, by writer Elizabeth Kolbert, to trying "to reassemble a Manhattan telephone book from pages that have been put through a shredder, mixed with yesterday's trash, and left to rot in a landfill".

Yet Pääbo has succeeded in this remarkable task, a story he recalls with striking frankness in Neanderthal Man: In Search of Lost Genomes, published by Basic Books later this month. Near the beginning, the 58-year-old geneticist says he was inspired to a life scientific by his biochemist father. It is only later that he reveals how odd was this paternal inspiration.

"I grew up as the secret extra-marital son of Sune Bergström, who won the 1982 Nobel prize for discovering prostaglandins," he says. Bergström's "official" family knew nothing of the existence of Pääbo and his mother, the Estonian chemist Karin Pääbo, with whom the Nobel laureate had had an affair. "He would only visit us on Saturdays," says Pääbo. "It was pretty weird with hindsight."

Bergstrom died in 2005. "It was only then my half-brother learned about me," says Pääbo. "Fortunately, he adjusted and we get on all right."

Pääbo studied medicine and later biochemistry at Uppsala University, where he began working with researchers who studied DNA to understand its relationship to disease. "They were interested in its role in living people but I wanted to know what it could tell us about ancient people. Were they closely related to people today, for example?"

Pääbo consulted textbooks but could find no references to DNA being extracted from dead tissue. No one had really thought about it. So, in the summer of 1981, he launched his career in ancient forensics, although he kept his work secret to avoid incurring the displeasure of his professor for indulging in "frivolous" activities. "I bought a piece of liver and stuck it in a laboratory oven, heated to 50C, for several days," he says. "Fortunately, the odour of putrefaction never reached my professor." Pääbo broke open the liver, by now hard, black and dry, and managed to extract DNA from it.

The DNA had been broken down into small sections only a few hundred nucleotide pairs long, compared with strips measuring thousands of nucleotide pairs that are typical of DNA extracted from fresh tissue. Nevertheless, there was sufficient DNA in his sample to back his belief that if it was extracted from old bones, genetic material could be used to create an entirely new way to probe the past: not through studying skulls, spears or hand-axes, but by analysing our predecessors' shredded biological leftovers.

Pääbo then tried to extract DNA from several Egyptian mummies (he admits to a particular fascination with ancient Egypt) and managed to isolate a short segment from the 2,400-year-old mummy of an infant boy. He wrote up the results for Nature in 1985, which the journal made its cover story, an astonishing achievement for a fledgling researcher.

Pääbo's research suggests that humans mated with Neanderthals following the exodus from Africa. Photograph: Jose A Astor/Alamy

A job at Munich University followed and in the early 90s Pääbo turned to a more ambitious project: Neanderthal DNA. The first remains of these ancient folk had been dug up in the 19th century in the Neander Valley, in Germany, and are now reckoned to belong a species of humans that became extinct more than 30,000 years ago. Most palaeontologists believe Neanderthals were overrun by genetically distinct modern humans when we emerged from our African homeland and spread round the world. Pääbo decided to investigate. "Neanderthal DNA seemed like the coolest thing imaginable to me," he says. (Cool is a common word in the Pääbo lexicon.)

Not surprisingly, few museum curators jumped at the chance to give Neanderthal bones from their collections to a bunch of stripling geneticists who wanted to saw lumps off them. It took several years of diplomacy before Pääbo was given an arm bone from the original Neander Valley skeleton, from which he extracted a 3.5g sample of whitish bone.

Pääbo tried to isolate DNA from its mitochondria – the tiny power packs that provide energy for our tissue and that are quite separate from the nuclear DNA that dictates the workings of the rest of an animal's body. Mitochondrial DNA is much shorter and more plentiful and so would be a better initial target, he reckoned. The ruse paid off and Pääbo isolated several DNA sequences that were clearly different from those of any human beings living today. The story made headlines round the world.

Then came the call from the Max Planck Society and an offer to join leading biologists, evolution experts and palaeontologists to establish a world-class anthropology centre in Leipzig. A substantial chunk of the institute's £20m construction budget would go to purchase new-generation sequencing machines that had already transformed the human genome project. Fired by his early success, Pääbo announced, in 2006, his group would sequence a full Neanderthal genome made of nuclear DNA within two years. In the end, the project was beset by tribulations – contamination, dastardly tricks by rival geneticists, dwindling supplies of Neanderthal bone – and Pääbo was more than a year late in completing the project.

His results provided a shock for both researchers and the public. When he compared his newly created Neanderthal genome with those of modern humans, he found a small but significant overlap in many of them. About 2% of Neanderthal genes could be found in people of European, Asian and far eastern origin. People from Africa had no Neanderthal genes, however. "This was not a technical error of some sort," Pääbo insists. "Neanderthals had contributed DNA to people living today. It was amazingly cool. Neanderthals were not totally extinct."

Most scientists, including Pääbo, now account for this result by arguing that modern humans – when they first emerged from Africa – encountered and mated with Neanderthals in the Middle East. Their offspring carried some Neanderthal genes and as modern humans swept through Asia and Europe they carried these genes with them.

The revelation that many humans possess Neanderthal genes fascinated the public. Dozens of individuals have since written to Pääbo claiming to be full Neanderthals. Intriguingly, nearly all of them have been men. The only women who wrote did so to say they thought their husbands were Neanderthals. "I think that says a lot about our image of Neanderthals," says Pääbo.

Just what that input of Neanderthal DNA has done for Homo sapiens's evolution is less clear. Pääbo speculates that changes in sperm mobility and alterations in skin cell structure could be involved. In addition, US researchers have recently proposed that Neanderthals passed on gene variants that may have had a beneficial effects in the past but which have now left people prone to type 2 diabetes and Crohn's disease. "This is work that is going to go on for years," he adds.

DNA from this fingertip, found in a cave in Siberia, identified a new human species, the Denisovans.

Pääbo's Neanderthal work is a towering scientific achievement. Yet the geneticist managed to top it in 2010 when his team announced that they had sequenced DNA from a tiny fragment of finger bone found in the Denisova cave in Siberia, and had shown it belonged not to Neanderthals as they had expected but to an entirely new human species, the Denisovans. More closely related to Neanderthals than modern humans, this ancient human "was the first new form of extinct humans to be described from DNA sequence data alone, in the total absence of any skeletal remains," Pääbo says.

This last point is intriguing. We have a fully sequenced Denisovan genome, thanks to Pääbo, but no idea what a Denisovan looks like. (A bit Neanderthalish is his best guess.) Thus Pääbo has taken his brainchild, the study of ancient DNA, and shown first, that it is a useful adjunct in the study of human origins; but now, and more importantly, he has developed his techniques to a state where they have become the prime means for discovering and defining new human species.

"Obviously there is a limit to how far back we can go in time with this approach," he says. "DNA fragmentation is a real issue. Nevertheless, I think one day that we will be able to sequence DNA that is 500,000 years old – and possibly a million years. And who knows what that will throw up."