Scientists made a remarkable discovery at Trou Al’Wesse in Belgium earlier this year. Inside a cave that overlooks the Hoyoux river they found clear evidence it had been occupied by Neanderthals tens of thousands of years ago. Yet the cave contained no skull fragments, no teeth – nor any other skeletal remains of this extinct species of human being.

The team, from the Max Planck Institute for Evolutionary Anthropology in Leipzig, were sure of their ground, however. Their genetic analysis of soil samples, scraped from the cave floor, had pinpointed the presence of Neanderthals through that most definitive of biological markers: their DNA.

In other words, without digging up a bone or a molar, the team, led by geneticist Matthias Meyer, had found – merely by studying a few microscopic strands of DNA – that tens of thousands of years ago Neanderthals had sheltered at Trou Al’Wesse. It was the scientific equivalent of “extracting gold dust from the air”, as one researcher put it.

Such hyperbole is understandable. The Trou Al’Wesse sediments would have been packed with DNA from plants, bacteria and other cave animals that had accumulated over millennia – as well as possible contaminating genetic material from the scientists themselves. Yet the Leipzig group, whose work was reported in Science in April, was able to pinpoint the few invisible scraps of Neanderthal DNA that had lingered there and enrich this material until they had enough to study its makeup in detail, a feat they later repeated at several other caves in Europe and Asia.

“We don’t know what was the exact source of this Neanderthal DNA,” Meyer told the Observer. “It could have come from Neanderthals who bled, or sweated, or left urine or faeces in the cave. However, once these cells had broken open, their DNA would have spilled out and would have become bound to minerals in the soil, where they were preserved.”

This is the genetic equivalent of uncovering the great library of Alexandria Johannes Krausse, scientist

Meyer’s project is an example of the astonishing advances that have been made in studying ancient genomes. Apart from detecting the presence of Neanderthals and other ancient people at sites devoid of any other remains, researchers are also using these techniques to uncover ancient population movements, pinpoint previously unknown human species, track the evolution of human illnesses and uncover the sources of human creativity. A new window has been opened on to our past.

“This is the genetic equivalent of uncovering the great library of Alexandria. All we have to do now is to learn how to read what we have found,” said Johannes Krausse, of the Max Planck Institute for the Science of Human History in Tubingen.

The study of ancient genomes began 20 years ago when scientists first developed techniques for extracting DNA from fossils and for creating sufficient copies of that genetic material to allow them to study and characterise them. This week leaders in the field will gather at the Wellcome genome campus in Cambridgeshire to mark the anniversary and discuss key recent developments. These will include work by researchers who have followed up the discovery that Neanderthals, who evolved separately from modern humans for more than half a million years, later interbred with us on several occasions as we emerged from our African homeland.

Other researchers will outline recent studies of the Denisovans, a mysterious people whose existence was revealed only when scientists extracted DNA from a tooth and a few bone fragments from the Denisova cave in Siberia and found it belonged to a previously unknown species of ancient human. Only a finger bone and three teeth of these people have ever been found, yet we know from genetic studies that Denisovans also interbred with modern humans. Their descendants, carrying small amounts of Denisovan DNA, then went on to settle in Melanesia and Australia thousands of years ago.

“Essentially, we are finding – thanks to DNA studies – that our relatively simple picture of human evolution was insufficiently detailed,” said Professor Chris Stringer of the Natural History Museum, London. “It is now clear there was a lot more interbreeding between ancient species, including early Homo sapiens and others, and that there was a lot more movement of populations both in the distant past – and relatively recently.”

Q&A What is DNA? Seven key facts Show Our genes are made of deoxyribonucleic acid, DNA, which forms double helix strands inside the nuclei of our bodies’ cells.

DNA contains information that is passed from generation to generation and directs the development of our bodies. Scientists began studying ancient DNA 20 years ago when Svante Pääbo used gene amplification techniques to extract and analyse genetic material from Neanderthals. At first only fragments of Neanderthal DNA could be studied, but by developing techniques to piece together small overlapping fragments, it was possible to recreate the entire set of genes, or genome, of a Neanderthal. Subsequent research showed that most people outside Africa contain small numbers of Neanderthal genes, the result of interbreeding between the two species as modern humans emerged from their African homeland about 70,000 years ago. In 2010 scientists discovered the genome of a completely new human species in bone and tooth fragments found in the Denisova cave, Siberia. These people are now known as Denisovans. Research published last year suggests that modern humans interbred with both Neanderthals and Denisovans on numerous occasions over the past 250,000 years.

One intriguing discovery is provided by the identification of the Ust’-Ishim man, a 45,000-year-old male whose remains were found in Siberia. DNA taken from his thigh bone indicates he was a member of Homo sapiens who possessed a distinctive genetic lineage that has since disappeared entirely from the human gene pool.

And a similar fate seems to have affected the Oase people of Romania. Named after a specimen found at Peştera cu Oase, Romania, this population of Homo sapiens contained fairly high amounts of Neanderthal DNA from recent interbreeding but also died out without leaving any further trace in the biological record.

In contrast to these lost lineages, scientists have also uncovered evidence that about 15,000 years ago, as the first farmers were developing in the Middle East, a group of people known as basal Eurasians added their DNA to the gene pool of these early agriculturists. Yet no one knows who these people were. The ghostly imprint of their genes in modern human DNA is the only evidence we have so far of their existence.

A major player in uncovering this picture of mysterious, shifting populations is the Swedish researcher Svante Pääbo, who led much of the effort to sequence the Neanderthal genome and the discovery of the Denisovans. Like most others in the field, he too has been startled by the rate of progress.

“I now think it is possible that we could sequence the genes of people and animals that lived up to 1 million years ago,” he said. “Already we have isolated genes from a horse that was 700,000 years old – though it helped that it was preserved in permafrost. However, if you had asked me 20 years ago what the limit would be I would have said we would be lucky to go back 100,000 years.”

The Denisova Cave in the Altai Mountains, southern Siberia, where evidence of a new species of ancient human was found. Photograph: Bence Viola/PA

The insights gained through the study of ancient genomes go far beyond the study of our past. “There is a great deal we can learn about modern humans from this work,” said Tony Capra of Vanderbilt University in Nashville. “For example, we know that around 2% of the genomes of modern humans is made up of Neanderthal DNA. However, some parts of our genomes are noticeable because they never contain Neanderthal DNA. That indicates these sections contain genes that are crucial to our success as a species. Individuals who inherited Neanderthal versions of these sections of genomes simply died out. That has given us a key focus to study these parts (of the genome) very closely – which is what we are doing – to discover how they brought us to global dominance.”

All of which suggests the study of genes is going to be of growing importance in understanding our species and that research on ancient bones will become a sideline of little interest. Why dig for years to find skeleton fragments when you can simply sift soil samples to reveal the presence of ancient DNA? It is a point stressed by one of the Wellcome conference’s organisers, Dr Chris Tyler-Smith of the Sanger institute. “I would advise any young scientist interested in this to give up the struggle to find bones of ancient humans and instead study ways to extract DNA from soil,” he said. “You will get a lot more science done.”

Not surprisingly, many other scientists disagree. “We wouldn’t have the Neanderthal genome today if palaeontologists hadn’t uncovered their skeletons and revealed the fossilised bones which contain their DNA,” said Pääbo. Stringer also disagrees. “Ancient genome studies are a magnificent addition to the tools we use to study our past. But we still need the physical evidence to tell us what they looked like. How they lived. Whether they buried their dead or painted caves. And there are many human and human-like species for which we don’t have DNA. We need to get the whole picture.”