A distant relative of ours? This reconstruction of a Neandertal child is based on a specimen discovered in Gibralter in 1926. Image: Wikimedia Commons

(PhysOrg.com) -- In the sometimes opaque world of statistics, Alan R. Templeton, Ph.D., professor of biology in Arts & Sciences at Washington University in St. Louis, has found that it's good to know your ABCs.

Templeton, with a doctorate in human genetics and a master's in statistics, has determined that a recently published genetic analysis of deep human DNA evolution is mathematically erroneous and formally illogical.

The flaws of the analysis are due to the incorrect application of a statistical method known as approximate Bayesian computation (ABC), which led Nelson J. R. Fagundes of the Universidade Federal do Rio Grande do Sul, in Porto Alegre, Brazil, and colleagues to support the validity of the controversial “Out of Africa” replacement hypothesis in a 2007 paper.

This hotly debated hypothesis claims that modern humans emerged out of Africa thousands of years ago and replaced or wiped out existing human populations. Templeton has done an acclaimed 2005 analysis that disputes this model, showing instead a trellis relationship between different human populations, supporting gene “admixture,” or intermingling.

Templeton got the notion to re-address the Fegundes paper when he noticed claims in a student’s paper using Bayes factors that were just too good to be true. Researching Bayes factors in the primary statistics literature, he found a 1999 paper that the probabilities generated by Bayes factors can be incoherent (result in conclusions that violate logic). He then re-read the Fegundes paper.

“When I first read the paper, I thought something was wrong with it, but I’ve got to admit I didn’t see the incoherence,” Templeton says. “I just saw these probabilities, and they didn’t make any sense to me, but I couldn’t quite pin it down. As soon as I read the 1999 paper, I went back to the 2007 one, and I saw that it was massively incoherent. It arrives at probabilities for different models of human evolution that violate the constraints of formal logic.”

Templeton published his analysis, “Coherent and incoherent inference in phylogeography and human evolution,” in a March 22 early addition online issue of PNAS.

In this paper Templeton points out that two models Fegundes examined, the out-of-Africa replacement model and the assimilation model are not independent of one other. The replacement model is just a special case of assimilation model with interbreeding falling to zero.

The problem is says, is that the authors found the probability of the replacement model being true, which is the special case, to be three orders of magnitude greater than the probability that the assimilation model, which is the general case, is true.

“These probabilities are in the wrong direction because they used a technique that was not designed to test nested models and used it on nested models,” Templeton explains. “I show in my paper that the fundamental equation that they used was logically incorrect and mathematically incorrect whenever you have logically overlapping models. In every case where you apply ABC to overlapping models it will give you a mathematical error.”

Templeton is not saying that Bayes factors or ABC cannot be used coherently. In fact, when it’s run on the simulated model B, a general model that is not nested, or not “the special case,” you get a coherent conclusion.

“The ABC method can be used legitimately to test nested hypotheses, but you can’t use the mathematically incorrect equations they use by simulating these models as if they were separate logical entities,” he says. “You can do the general model and then look at the parameter that defines the special case, and when you do that you have a complete reversal.”

Surprise of surprises, when the coherence is corrected in the Fegundes model, “it’s completely compatible with my earlier trellis model,” Templeton says.

Explore further Technique traces origins of disease genes in mixed human populations