Bernie's Basics

Stats without maths

You don't need to know your Chi square from your chai latte to make sense of statistical headlines. Just get your head around the lingo and some Year 5 maths.

There are two problems with the language of statistics, plus or minus one:

it's full of regular sounding words like 'significant' and 'likely', but their statistical meanings are far from regular

statistics are usually given as a percentage, but percentages are like shorthand. They never tell the full story, so there's always room for dodgy interpretations, and

because they sound so precise and mathematical we expect statistics to give definite yes or no answers, or prove something is true but they never can. Statistics isn't about certainty. It's about calculating the odds of something, and any gambler will tell you there's no such thing as a sure thing.

It's the lack of certainty of statistics that makes the mathematical back-up and the precise "I, Robot" style language so important. Researchers who use statistics have to master both. Luckily the rest of us only need to understand a few catch-phrases and a couple of rules to make sense of what the experts are saying.

^ to top

You say significant, I say 95 per cent likely to have not happened by chance

Pernickety isn't a statistical term, but it should be. Even the most ordinary sounding words have very particular mathematical meanings in statistics.

When a statistician says it's "very likely" something will happen, they don't just mean they're fairly sure. They literally mean there's a 90 to 99 per cent chance it will go ahead. Not a 72 or 87 per cent chance (that would just be 'likely'). And not 99.5 per cent ('virtually certain').

At least with 'likely' the basic meaning of the word doesn't change in statistics. But when a researcher says their findings are 'significant', it doesn't mean they've stumbled on something important like cold fusion or a cure for split ends. It just means there's a 95 per cent chance that whatever link or pattern they've seen in their data is real, and not just some unusual but meaningless lump.

"There's a 95 per cent chance this thing is real" is not the giddy stuff of headlines, but statistical significance gives researchers an idea of whether something might be worth looking into further.

Even so, almost five per cent of the time the finding will end up not being real after all: just because it's unlikely that it's not real doesn't mean it's definitely not real. So about one of every 20 'significant' findings that are announced or published will end up being duds. Worth remembering come news time.

If we're prone to overestimating the significance of the term significant, when it comes to statistical relationships we're really at risk of getting carried away.

^ to top

Relationships — a great source of confusion

We've all seen enough TV to know that it's hard to prove guilt beyond doubt. But that's nothing compared with how hard it is to prove causation between two statistically related factors. Ask anyone who studied the relationship between smoking and lung cancer.

By 1950 strong evidence from enormous, well-designed studies clearly showed that there was a significant relationship between the two — the level of lung cancer among smokers was significantly higher than among non-smokers. That kind of relationship, where one increases with the other, is called a correlation.

The researchers rightly pointed out that just because there was a clear relationship between smoking and lung cancer didn't mean smoking caused cancer. (You'd see the same relationship if lung cancer caused smoking, or if both smoking and cancer resulted from the same common cause). It was 1964 before the US Surgeon General finally came out and said smoking was the main cause of lung cancer, and that heavy smokers were 20 times more likely to develop it than non-smokers.

So why was it so hard to go from correlation to cause? Because it's incredibly hard to prove that one thing definitely causes another unless you can rule out all the other factors that might play a role. The only way to rule those factors out is to control for them. And the double-blind randomised controlled trial (RCT) used in testing new drugs (where no one knows who's getting the drug and who's getting the placebo) is the accepted finger pointer when it comes to establishing cause.

The RCT is great for testing drugs and treatments, but it's more than a little impractical for establishing cause between things that don't involve a pill. How do you make a placebo cigarette for the control group for starters? And how ethical is it to get someone to take up smoking if you already suspect it will up their risk of developing cancer? The only option for smoking researchers — and for most public health research today — was to use the enormous experiment that was already going on. They compared the health outcomes of millions of smokers with non-smokers, trying to match all other characteristics that could play a role in developing cancer (like age, gender, ethnicity, socioeconomic status) so they could be as confident as possible that it was the smoking that was behind the lung cancer.

Epidemiological studies like these can never provide 100 per cent proof of cause for the same reason that climate scientists will never be able to say it's 100 per cent certain that we're causing climate change. If there's one thing for certain, it's not statistics. The closest we'll get is a "virtually certain" more than 99 per cent likely.

And that can only happen if you've got a swag of well designed studies showing the same result, and the 'cause' you've found makes sense, ie

if smoking causes cancer, the smoking has to come before any sign of cancer

heavier smokers have higher rates of lung cancer (called a dose-response relationship)

no other factor can explain the relationship between smoking and cancer

the connection between inhaling hot carcinogens and developing lung cancer itself makes sense.

So it's not surprising that most research doesn't establish cause between two statistically related things. What epidemiological studies do find is the relationships themselves — everything from burnt toast to red wine has been linked with cancer risk. And risk is the last bit of lingo that we have to master to cut through the headlines.

^ to top

A tale of two risks

You wouldn't be the first person to crack another bottle of red because it's supposed to be good for your ticker. But headlines like "reduces risk of heart disease by 20 per cent" are misleading for most of us.

That's because in the world of health and medical research there are two kinds of risk, and they paint very different pictures.

The first kind, absolute risk, says how likely it is that something will happen to any one of us within a certain period of time. Like getting heart disease.

A middle-aged non-smoker with regular blood pressure and cholesterol levels like me has less than a five per cent risk of getting heart disease in the next five years. That means for every hundred women like me, fewer than five of us should have a heart attack or stroke within five birthdays. If I was a bloke the odds would be a bit higher — less than nine per cent.

If I was a smoker with the same blood pressure and cholesterol levels I've got now, my risk would be somewhere between five and 15 per cent. That's a jump in absolute risk of up to 10 per cent, but that's not what the headlines will say. They'll run with "Smoking increases heart disease risk by up to 200 per cent". And they wouldn't be wrong — they'd just be misleading, because they're talking about the other kind of risk, relative risk.

Relative risk is probably the most confusing thing about research findings for those of us not in the health game. And its definition doesn't shower it in clarity: it's the ratio of absolute risks expressed as a percentage. So in the smoking example the 10 per cent increase in our absolute risk becomes a 200 per cent increase in relative risk (10/5 x 100). If the idea of relative risk is to freak us out, it works. If it's to help us make informed lifestyle decisions, it's a bit on the lies, damn lies side of transparent.

Of course, come "another pinot?" time, it's everybody's favourite kind of statistic.

Thanks to Associate Professor Lyndal Trevena, Sydney School of Public Health, University of Sydney

^ to top