It's now clear that our ancestors, upon leaving Africa, interbred with the Neanderthals and Denisovans, archaic humans that occupied Europe and Asia. The first offspring of those pairings would have had equal amounts of their two ancestral legacies: one chromosome in each pair would be modern human, the other archaic. But most of that archaic legacy is now gone, down to about two percent from the initial 50. What happened?

People have been considering a number ideas. These range from the simple—a large and growing modern human population simply swamped the archaic genes—to more complex situations such as limited reproductive compatibility between the two genomes. But a new paper suggest something in between the two. Many Neanderthal genes might be slightly deleterious, the research suggests, but the Neanderthal populations weren't large enough to get rid of them.

It's difficult to understand the fate of Neanderthal genes, since we don't have any sense of how well the offspring of these matings did. All indications are that the instances of successful interbreeding were rare (with success here defined as offspring that passed on their genes as well). Since modern humans were expanding into new territories and increasing their population accordingly, it may simply be that the Neanderthal genes were diluted as their carriers continually mated with people who were largely or entirely modern.

But it's also possible that the answer lies in the genome. There are clear indications that some of the Neanderthal genes may be adaptive, helping modern humans in what was a very different environment from the Africa they left. But it could also be that either the chromosome structure or genes of Neanderthals limited the fertility of the hybrids. This would greatly accelerate the dilution process described above. Alternatively, it could be that some Neanderthal genes affected the health of the hybrids, making them a bit less fit than humans with a more thoroughly modern genome.

It's this last possibility that the new paper looks at. Several researchers at the University of California, Davis built a mathematical model to see what might happen to Neanderthal genes under a variety of scenarios. These included different degrees of dilution by modern human genomes, as well as different degrees of evolutionary selection against Neanderthal genes.

The versions of the model that produced the best matches to current genome suggests that relatively rare gene variants (about one difference in every 10,000 bases) that were weakly selected against by evolution. The strength of the selection against this is, in the authors' words, "very low." But given tens of thousands of years, it's enough to reduce the amount of Neanderthal DNA left around, although only by an average of 56 percent. In keeping with this suggestion, the amount of Neanderthal DNA also appears to be lower in gene-rich regions.

If the DNA were problematic, how did Neanderthals manage to spread throughout Europe and Asia in the midst of a glacial period without getting rid of it? The Neanderthal genomes we've looked at are very low-diversity, meaning that there's not a lot of differences among them. This suggests the effective Neanderthal population size was low, and evolution needs a significant effective population size to work. This is for purely practical reasons. To get rid of a harmful mutation, you have to replace it with a better variant; when the population isn't diverse, there's a good chance that better variants aren't around.

The authors calculate that, if the same level of selection applied in Neanderthals, the genetic variants would be neutral in the smaller population. In other words, they'd be invisible to evolutionary selection within the Neanderthal population. But they wouldn't be invisible overall. The authors calculate that the genetic load would radically reduce Neanderthal fitness as compared to modern humans, which could help explain why they were rapidly displaced as modern humans moved in.

In any case, the selection against Neanderthal DNA isn't strong enough to have completely eliminated the presence of their genes (as opposed to non-coding DNA) in modern humans. The authors estimate that there are about 7,000 deleterious Neanderthal alleles presently circulating in Eurasian populations, with the average individual having about 100 of them. So, it may be possible to actually test this mathematical model using more concrete data.

PLOS Genetics, 2016. DOI: 10.1371/journal.pgen.1006340 (About DOIs).