Modern humanity's ancient cousins, the Neanderthals, lived in small groups that were isolated from one another, suggests an investigation into their DNA. The analysis also finds that Neanderthals lacked some human genes that are linked to our behavior. (Related: "Why Am I Neanderthal?")

In recent years, experts in ancient DNA have mapped out the genes of Neanderthals, a species of human that vanished some 30,000 years ago. These gene maps have revealed that many modern people share a small part of their ancestry, and a small percentage of their genes, with those early humans.

Now moving beyond ancestry, researchers are comparing these ancient gene maps to those of modern humans. The comparisons may point to genes that make us uniquely human and uncover links to the origins of genetic ailments.

Compared to Neanderthals, humanity appears to have evolved more when it comes to genes related to behavior, suggests a team headed by Svante Pääbo, a pioneer in ancient genetics at Germany's Max Planck Institute for Evolutionary Anthropology. Their study was published today in the Proceedings of the National Academy of Sciences.

They note in particular that genes linked to hyperactivity and aggressive behavior in modern humans appear to be absent in Neanderthals. Also missing is DNA associated with syndromes such as autism.

"The paper describes some very interesting evolutionary dynamics," said paleoanthropologist John Hawks of the University of Wisconsin at Madison.

The Neanderthal genes suggest that sometime after one million to 500,000 years ago, Neanderthal numbers decreased and the population stayed small, Pääbo's group determined. A small population size would have been bad news for Neanderthals, Hawks said, because it would have meant that "natural selection had less power to weed out bad mutations."

Ancient Answers

Pääbo and colleagues looked at the genes of two ancient Neanderthals, one from Spain and one from Croatia. They compared the DNA of those individuals to that of a third Neanderthal who had lived in Siberia and whose DNA had been analyzed in an earlier study, and to the DNA of several modern humans.

"We find that [Neanderthals] had even less [genetic] variation than present-day humans," Pääbo said by email. Genetic diversity among Neanderthals was about one-fourth as much as is seen among modern Africans, he said, and one-third that of modern Europeans or Asians.

The comparison also showed that modern people possess genes linked to heart health and metabolism that were absent in our ancient cousins. We also have genes for skin and hair color they lacked. "These mutations may contribute to differences in pigmentation among present-day humans," Pääbo and colleagues wrote in the PNAS study.

Hawks cautioned, however, that comparing ancient humans to modern ones is a bit misleading, "because humans today have a recent history of massive population growth."

That recent population growth obscures the signal of gene differences that are part of our ancient heritage, when people lived lives more like the Neanderthals', and ones that sprang up only since the advent of farming.

If scientists want to know what really separates Neanderthals from Homo sapiens, "we need to figure out how much of today's pattern comes from ancient hunter-gatherers, and how much comes from our recent adaptation," he said.

The Pääbo team also compared the Neanderthal and modern human DNA to that of a Denisovan, another early human cousin species that lived in Siberia as recently as 40,000 years ago and is known only from genetic maps and a few scraps of bones.

The main genetic difference found in that comparison relates to the curvature of the lower back, according to the study. Essentially, Neanderthals looked a bit less sway-backed than do modern humans, Denisovans, and our other early ancestors.

View Images Neanderthals and modern humans have differences in genes linked to aggression. PHOTOGRAPHS BY JOE MCNALLY, NATIONAL GEOGRAPHIC

Ancient Questions

Humans, Neanderthals, and Denisovans shared a common ancestor more than half a million years ago. That connection, however, has been muddied by more recent interbreeding that likely took place among our ancient ancestors after early modern humans migrated out of Africa about 60,000 years ago.

Modern people of Eurasian descent owe about 2 percent of their genes to ancient mating with Neanderthals, and people of Melanesian descent owe perhaps 3 to 5 percent of their genes to Denisovans for the same reason. (Related: "Surprise! 20 Percent of Neanderthal Genome Lives On.")

But the story doesn't end with the mere presence of genes or mutations within them. The PNAS study found, for instance, genetic differences between Neanderthals and modern humans in genes linked to aggression. But the authors caution that "if [the genes] affected activity or aggression levels, it is unclear whether they increased or decreased such traits." They cannot say, for now, which species would have been the more aggressive one.

And there's also the question of how often any genes present were actually used, called gene activity. In a recent study published in the journal Science, Israeli researchers teamed up with Pääbo's group to look at gene activity in Neanderthals who lived 50,000 or more years ago. (Reas "The Case of the Missing Ancestor" in National Geographic magazine.)

That study showed how differences in the expression of genes shared by modern people and our ancient cousins led to Neanderthals possessing bigger hands, shorter arms, and a more burly build.

"The conclusions regarding skeletal change being greater along the Neanderthal lineage than the modern human one appear to be at odds with the usual view of Homo sapiens," said human origins expert Chris Stringer of the Natural History Museum in London. Compared to the robust-looking earlier human species that preceded Neanderthals, humans appear more changed by evolution, with lighter skulls and thinner bones.

Pääbo cautioned, however, against equating genetic change with evolutionary sophistication. "Clearly, Neandertals were not 'less evolved' than modern humans," he said. "They had their own history and evolution. They took another path, if you like."

Follow Dan Vergano on Twitter.