Well, well , well, what do we have here? Not THC, honest! Nick Cote/The New York Times/Redux/eyevine

Last month, the drinking water in a Colorado town was declared unsafe, because it had been contaminated by an ingredient from cannabis. It took two days to discover that this was not the case – a water test had turned up a false positive result. In fact, false positives are widespread in our everyday lives, and we seem to have an innate inability to get to grips with them.

The fuss in Hugo, Colorado – a state where cannabis use is now legal – began when a county employee administering a test for drug use decided to use the same kind of test on tap water, rather than saliva, in an attempt to rule out a false positive. When the water tested positive too, it was assumed the test kit was a dud. But when they tried a different kit from another manufacturer the result was the same – it appeared that there was THC, the primary psychoactive compound in cannabis, in the water.

Police then investigated the town’s well and found what they thought was evidence it had been tampered with. Residents were advised to switch to bottled water and restaurants were closed. But THC does not dissolve easily in water. If someone had wanted to contaminate the town’s water supply with this compound, they would have needed a huge – and expensive – amount of THC.


False confessions

By chance alone, we should expect things like this to happen. Any test will turn up a result that isn’t accurate every now and then, and we would expect this on occasion to happen in the same place multiple times. But our brains seem to have particular trouble handling these kinds of probability estimations.

When the FBI conducted further tests, it emerged that the initial findings had been false positives. False positives can be caused by anything from faulty test kits to contamination, or even what you eat – people can test positive for opiate drugs after eating poppy seeds.

In the US a $2 roadside drug testing kit that is widely used by police to identify suspected drugs has led to controversy because it produces false positives when it is exposed to more than 80 compounds, including common home cleaning products. There is no official expected false positive rate for the test, largely because accuracy can vary depending on who is using it, and how they administer it.

Police forces in the US tend to use these kinds of tests because the more reliable mobile chemistry sets they used to use were prone to leaking and causing harm. Disposable field kits started becoming widespread during the 1970s, but as far back as 1974 a study warned that such kits were not reliable enough to be the only type of testing evidence used in prosecutions. But a study by the investigative journalism organisation ProPublica suggests that individuals today are often pressured into false confessions on the basis of false positives from such kits. When a subject pleads guilty, their sample is less likely to be double-checked by a federal drug lab.

Screening struggles

Police forces aren’t the only people prone to picking up false positives – doctors have a real problem with them too.

One study has found that doctors often underestimate the likelihood of getting a false positive result during breast cancer screening. The researchers told the doctors that 1 per cent of women have breast cancer, and that such women are 90 per cent likely to test positive. The false positive rate was given at 9 per cent.

Can you work out from this information how likely it is that a woman with a positive test result has cancer?

The correct answer is that in this scenario, only about 1 in 10 women who test positive really do have breast cancer. If you got that right, you’re doing better than 79 per cent of the physicians who took part in the study, who put the answer at eight or nine out of 10.

The reason we find it so hard to understand false positive rates seems to be largely because we are easily confused by conditional probabilities – the likelihood of something happening, if a particular factor is a certain way.

Living with uncertainty

One strategy to get around this is to think about people in terms of whole numbers rather than in decimals and percentages – for example, cancer cases in 1000 women. The researchers found that with this presentation, 87 per cent of doctors picked the correct answer from a multiple choice of four options.

The growing realisation that the majority of “positive” results found in cancer screening are in fact false alarms, has resulted in claims that broad cancer screening is unethical and should be scrapped. False positives from cancer screening can result in harmful treatments such as unnecessary surgery and radio- and chemotherapy.

According to the researchers behind the screening study, we are inherently prone to an “illusion of certainty”, having an emotional need for certainty where it simply does not exist. We just can’t get our heads around the fact that the error rates of scientific tests are often far higher than the incidence rates of the things they’re detecting – a phenomenon dubbed the false positive paradox. This can lead us to fail to recognise false positives, and draw the wrong conclusions. It’s a fact doctors, police, patients and politicians would do well to understand better.

Read more: Thousands of fMRI brain studies in doubt due to software flaws