What are the odds that he deserves the cuffs? (Image: Erika Kyte/Photonica/Getty) Lab survey: the state of DNA analysis

Read more: Fallible DNA evidence can mean prison or freedom

CHARLES RICHARD SMITH has learned the hard way that you can prove almost anything with statistics. In 2009 a disputed statistic provided by a DNA analyst landed him with a 25-year jail sentence.


Smith was convicted of a sexual assault on Mary Jackson (not her real name) in Sacramento, California, which took place in January 2006. Jackson was sitting in a parking lot when a stranger jumped into her truck and made her drive to a remote location before forcing her to perform oral sex on him. When police arrested Smith and took a swab of cells from his penis, they found a second person’s DNA mixed with his own.

The DNA analyst who testified in Smith’s trial said the chances of the DNA coming from someone other than Jackson were 1 in 95,000. But both the prosecution and the analyst’s supervisor said the odds were more like 1 in 47. A later review of the evidence suggested that the chances of the second person’s DNA coming from someone other than Jackson were closer to 1 in 13, while a different statistical method said the chance of seeing this evidence if the DNA came from Jackson is only twice that of the chance of seeing it if it came from someone else.

How can a single piece of DNA evidence generate such massive differences in the statistical weight assigned to it? Last week, a New Scientist investigation showed how different forensic analysts can reach very different conclusions about whether or not someone’s DNA matches a profile from a crime scene. This week we show how, even when analysts agree that someone could be a match for a piece of DNA evidence, the statistical weight assigned to that match can vary enormously.

“Usually DNA evidence is pretty strong,” says David Balding, a statistical geneticist at University College London, whose calculation puts the lowest probability on the link between Smith and Jackson. “My point is that the number juries are provided with often overstates the evidence. It should be a smaller number.”

The odds juries are provided with often overstate the evidence

There are several types of statistic that analysts can attach to DNA evidence. In basic cases involving a large amount of DNA from a single person, you can simply calculate how common their profile is in the general population- this is called the random match probability (RMP). However, the RMP becomes problematic when looking at mixed or degraded samples of DNA, where part of a person’s DNA profile may be missing or hidden by another person’s DNA.

For this reason many labs will use a different statistic when interpreting mixtures, such as “random man not excluded” (RMNE) or the “combined probability of inclusion or exclusion” (CPI/E). These calculate the odds that DNA in a mixture is from a random person rather than the person you’re interested in.

But this approach by no means solves the problems. In Smith’s case, two of the statistics given- 1/95,000 and 1/47- were the result of RMNE or CPI calculations, while the 1/13 statistic was a variation on these.

A DNA profile consists of a series of peaks relating to specific locations on the chromosomes, called loci. In a standard profile there should be peaks indicating two genetic sequences, or alleles, at every locus- one from each parent. However, in mixed profiles or when only small amounts of DNA are present, it can be difficult to work out which alleles came from whom, and even to detect whether certain alleles are present (New Scientist, 14 August, p 8).

In the Smith case, the sample containing another person’s DNA showed alleles at seven out of a possible 15 loci, but at four of these loci, the alleles matched those of both the victim and the defendant. “The 1 in 95,000 figure in effect treated these alleles as full-weight evidence that the DNA came from the victim, ignoring the alternative possibility that the allele we saw could have been from the defendant,” says Balding. If the opposite position is taken, and these alleles are ignored, you come up with a figure closer to 1 in 13. “It’s a question of which loci you consider,” he says.

At present there are no firm guidelines on which alleles should be included in an RMNE calculation, meaning that different labs can come up with very different statistics.

Balding and Peter Gill of the University of Strathclyde in Glasgow, UK, who is a former chief analyst at the UK’s Forensic Science Service (FSS), have developed a different statistic, the likelihood ratio (LR), which attempts to take phenomena like missing alleles into account. “The likelihood ratio tries to do the right thing,” says Balding. “It asks how probable is this evidence under the prosecution case, and how probable is this evidence under the defence case? Then you take the ratio of the two.”

Useless at trial

In the Smith case, Balding calculated an LR of just 2, meaning that the DNA evidence is only twice as likely if it came from the victim as from someone else. Since there are many alternative possibilities for the source of the DNA, a 2 to 1 ratio means the evidence is so weak as to be virtually useless at trial.

Yet neither the LR of 2 nor the 1/13 statistic was presented in court. Smith was ultimately convicted on the basis of this DNA and other evidence. He is still in prison, and plans to appeal.

Using a likelihood ratio doesn’t necessarily mean that the defendant will get an easier ride. In a 2003 murder case in the UK, Balding’s LR found that the evidence was 10 million times more likely to fit the prosecution’s case than that of the defence (Forensic Science International: Genetics, DOI: 10.1016/j.fsigen.2009.03.003).

“All evidence is worthwhile, it’s just a question of can you evaluate it fairly?” says Balding.

These cases are not isolated incidents. A recent study by John Butler at the US National Institute of Standards and Technology (NIST) in Gaithersburg, Maryland, found widespread variation in the statistics reported by individual laboratories. When he gave the same DNA evidence to 69 different US labs and asked them to provide conclusions about whether or not the suspect was a match, some labs reported RMP, others reported RMNE, while others gave no statistic at all. “There was a difference of about 10 orders of magnitude in terms of the statistical results that were obtained on the same samples,” says Butler.

Even among the labs using the same statistic- the RMNE- there were differences in the figure they came up with, depending on which alleles they chose to include or discard. What it could mean is that a jury presented with evidence from one lab is told that the chance of a match is around 1 in 100,000, say, while a different lab might say it is 1 in a quadrillion (1015).

Throwing evidence away

Our own survey of DNA laboratories backs up these conclusions (see chart). We asked 19 labs around the world which statistics they report when dealing with complex DNA mixtures. Six said that they reported RMP, six reported RMNE, while two said they reported LR. Five labs said they either report no statistics or a CPI/E. The type of statistic reported even varied within the same US state.

The International Society for Forensic Genetics recently issued guidelines recommending the use of the LR when dealing with complex mixtures. Even so, few labs have taken it up. The Institute of Environmental Science and Research lab in Auckland, New Zealand, which performs DNA analysis for the New Zealand police, is one of the few to do so. “Likelihood ratios are more complex, and they’re harder to present in court,” says John Buckleton, head of ESR’s biology lab. For this reason, introducing them can require training for judges to ensure they can accurately explain the statistic to jurors.

Even the UK’s FSS, where Gill worked for 25 years, has not fully implemented LRs. “They report LRs, but they’re not necessarily incorporating things like the probability of [missing alleles],” says Gill. “By not using the proper statistical methods, you’re just throwing evidence away.” When New Scientist contacted the FSS, we were told that it does evaluate DNA evidence using LRs. “In addition,” said a spokesperson, “we use the replication method of analysis when dealing with low levels of DNA, which means that the scientist reports the alleles that can be replicated.”

Earlier this year, the Scientific Working Group on DNA Analysis Methods (SWGDAM), which provides guidance to US forensic labs, issued its own recommendations regarding the use of statistics for DNA mixtures. It also proposes LR as a suitable statistic, as well as providing stronger guidance on how RMNE and RMP should be calculated. SWGDAM also insists that analysts must provide a statistic whenever they claim that someone’s DNA might be included in a mixture. “There are some labs that are just reporting that they think it’s a match- in their opinion,” says Butler. “That’s a problem, because the jury says: ‘Oh, it’s DNA? It matches? Guilty.'”

Some labs give no statistics on a match, just an opinion. That’s a problem

Ultimately though, there will be cases where the DNA evidence is so complex that it is impossible to generate a reliable statistic. Buckleton admits that DNA analysts can come under pressure from prosecutors and police, as well as defence lawyers, to render DNA evidence useable. “I’m not sure that the pressure is to drive the numbers in any particular direction, but perhaps to be more certain than you should be at some points,” he says.

In really complex cases, analysts need to be able to draw a line and say “This is just too complex, I can’t make the call on it,” says Butler. “Part of the challenge now, is that every lab has that line set at a different place. But the honest thing to do as a scientist is to say: I’m not going to try to get something that won’t be reliable.”

When lawyers question DNA In 2007, Sean Hoey was cleared of involvement in the 1998 Omagh bombing in Northern Ireland, which killed 29 people. At the heart of the case against him was DNA on the bomb timers, which the prosecution alleged matched Hoey’s. It was the first time that defence lawyers had challenged DNA evidence in a UK court, and they were successful, arguing that DNA analysts were divided over the reliability of the technique used. In low-copy-number (LCN) testing, tiny amounts of DNA are amplified to generate a profile. Since then, much of the scrutiny surrounding DNA evidence has focused on the reliability of LCN testing, but some feel that this emphasis is misplaced. In the wake of the Hoey case, Belfast-based solicitor Peter Corrigan of Kevin Winters and Company has routinely sought access to the lab reports behind the DNA evidence presented in court, which has resulted in four successful challenges. “The underlying data had never been subject to any court scrutiny,” he says. “Defence experts were trusting that the scientists had interpreted the data correctly. This perpetuated the myth that DNA is infallible.” Peter Gill of the University of Strathclyde in Glasgow, UK, a former analyst at the UK’s Forensic Science Service, admits that there is a problem. “There’s a considerable lack of understanding, not just from the public, but from the judges and lawyers.” The problem is not confined to the UK. “In our experience, examination of the underlying data frequently reveals limitations or problems that would not be apparent from the lab report,” says William Thompson of the University of California, Berkeley, who acts as an expert witness on DNA. However, “forensic DNA analysts tell us that they receive requests [for DNA lab reports] from defence lawyers in only 10 to 15 per cent of cases in which their tests incriminate a suspect,” Thompson says. Even when the defence makes a legitimate challenge, the public rarely hears about it. In the UK, defence lawyers are granted access to DNA data on the condition that they only use it in the case in question. If the questioned evidence is dropped before it gets to court, this never becomes public, says Allan Jamieson of the Forensic Institute in Glasgow.