The following is taken in part from Episode 46 of our podcast ‘Skeptics with a K’, give or take the odd addition.

A generation of children ‘turn their backs on sport’ – so said the BBC recently. And they weren’t alone, with similar stories gracing the pages of the Daily Mail, The Independent and pretty much every other media outlet going. But I’ll focus on the BBC, because I respect them most. Moving on with the story:

A generation of British children are turning their backs on sport and physical activity, a survey suggests. The poll for British Triathlon and Tata Steel suggests 10% cannot ride a bike and 15% cannot swim.

Connoisseurs of my PR takedowns in the past will spot the brand names right there in paragraph two – British Triathlon and Tata Steel. The latter are a steelworking giant who sponsor the Tata Kids Of Steel – a community programme to drive kids into exercise, and in particular into the swimming, bike-riding and running that constitutes the triathlon, as promoted by British Triathlon.

Now, it’s worth pointing out at this point – just because the British Triathlon federation and its corporate sponsor Tata Steel have a vested interest in telling the world that children are no longer riding bicycles and swimming and generally triathlonning, it doesn’t mean the survey involved here is dodgy. But it does mean we should be treading a little carefully, and we should certainly be examining the claims being made perhaps a little more skeptically than if an entirely independent body were making the same claims.

As a brief aside at this point, it’s worth pointing out that the first thing I thought when I glanced over this story was ‘who are Tata Steel’ and ‘what have they got to do with sports’ – questions which were soon answered with a mild Google. These big businesses aren’t stupid, and I’d speculate that for every pound spent on this sports initiative, a corporate sponsor would see two pounds or more come back to them in either goodwill, reputational benefit, or convenient blind-eyes to some of the inevitably murkier elements of a large-scale industrial business.

Anyway, back to the BBC, and the story we’re being cautiously skeptical about, and here come the statistics:

The survey of 1,500 children aged six to 15 found almost a quarter (22%) had never run a distance of 400 metres. A third of the children questioned said they did not own a bike, while three quarters (77%) had a games console and 68% had a mobile phone of their own. In the week before the poll was conducted in March, just 46% had ridden their bikes and 34% had swam the length of a pool, but 73% had played a video game. And 15% of the children said they had never played sport with their parents.

What we have here is a reasonable smattering of statistics which appear to tell a story – our kids are fat lazy layabout couch potatoes, shunning sport and addicted to TV. In fact, many of those exact terms appear as headlines in the news coverage, with the all-important contentious quotation marks around the judgemental phrases, allowing the reporter to distance themselves from the derogatory terms and pretend that the judgements came directly from the data, rather from the sub-editor themselves. It might be worth pointing out that such sub-editing tricks are ‘pretty annoying’, that the sub-editors ‘lack the balls to stand by their editorial voice’ and are ‘probably allied with Satan’. See – because I used quote marks there, it made it seem like I was citing a source, rather than tossing out attention-grabbing insults.

Still, when you begin to look at the data, it’s clear the statistics don’t go nearly as far as the headlines would like, and certainly aren’t without potential biases – as we so often see when numbers are presented with no strong context to explain what they’re really showing. Without having the full results available, it’s impossible to rule out all manner of flaws – but, of course, the source data isn’t provided, and we’re left twiddling our thumbs if we want to work out what the numbers really mean. So, thumb-twiddling away, here are a few ideas about where there could be seen to be problems with the story – none of these are definitive, they’re just my ideas, but equally none of these can be ruled out given the meagre information we’re provided with, to back up what is a story of clearly widespread interest:

1) The data set we’re working with is a reasonably decent age range – ten years, at its most inclusive. If we’re looking at the behaviour of people in their twenties or people in their fifties, it’s a good basis to work from. However, when we’re looking at children, it becomes a huge range. The abilities of a child at 6 years old and at 15 years old are so different, they’re incredibly difficult to lump together. That’s not to say the information presented is worthless, but that taking broad brush answers from it without the ability to dig into some of the data leaves us exposed to over-extrapolate the storyline.

Consider this – at what age do you learn to swim? And how far and how confidently would you have to be able to swim to tick the ‘I can swim’ box in this survey? Depending on the definition, it might not be too surprising that a good portion of the 6, 7, 8 and even 9 year olds would fall at this hurdle. In fact, given that 10% of kids couldn’t swim, and the survey looked at 10 different ages, it would simply take all of the 6-year-olds being unable to swim to a level that ‘counts’, and you’ve already hit your 10%. In that scenario, 100% of kids aged 7-15 could be perfectly adequate and keen swimmers.

Similarly, at what age do you ride a bike, and what counts as being able to ride it – stabilisers on or off? If we said 6 and 7 is at the younger end to confidently ride a bike, then you could see a big chunk of your 15% right there. I’m not saying that’s definitely where all of the non-bike-riders are, but I’m saying that’s the kind of margins we’re working with here. Without being able to tell what percentage of each smaller age range were unable to do those activities, the stats don’t tell us an awful lot. Did you know, for example, that more than half of 6 to 15-year-olds don’t have a single passing grade in their 11+ exams? The GCSE pass rate in children aged 6 to 16 is shockingly low too.

2) We don’t know how representative the selection was – there’s nothing in any of the articles to confirm that the 1500 children surveyed were split evenly across the 10 ages. I’m not implying it wasn’t – we simply aren’t told this – but if it was skewed in the favour of the younger children (ie 300 6-year-olds and only 50 15-year-olds), then you’d see the information quality being potentially further questionable. Looking again at that stat about 22% of children never having run 400m, and you could see how a skewed data set could influence that.

On this point, at least, I managed to get clarification – after emailing the British Triathlon Federation for their source results (something I urge you all to do when you see a survey you’re unsure about), I was passed to their PR firm Clifford French, who confirmed via email that:

“…the spread of respondents was evenly distributed across the age group and the UK; as was the percentage of kids who couldn’t ride a bike and swim.”

So, presumably, we’re looking at 150 children in each age (give or take). Interesting, then, that the findings appear to show the same percentage of 15 year olds unable to swim or ride a bike as was found in the group of 6 year olds. I, for one, find this highly questionable – lacking the data to confirm, my skepticism is speculative, of course.

3) In fact, the 400m stat doesn’t even need a skewed data set to influence it – how many children know what 400m is? And how many 6-15 year olds would have run 400m without knowing how far 400m is, or noticing that they’ve ran that distance (playing football for example, or hide and seek, or just running for the sake of running, which I used to do as a kid).

4) What happens when you logically-reverse the stats? If you read in the paper that 90% of children can swim, and 85% can ride a bike, would you think that was shockingly low? Would it be headline-worthily low? I’m not so sure it would. But lead with the negative, and you create a story. After all, is it fair to say that “a GENERATION of kids are turning their backs on sport” – direct quote from the BBC there – simply because every tenth child can’t ride a bike?

5) Looking at the other stats, “In the week before the poll was conducted in March, just 46% had ridden their bikes”. Even ignoring that March may well have been quite a chilly month, not conducive to a nice bike ride, there are still questions to be raised. It might seem obvious, but is this figure an answer to the question ‘Did you ride a bike in the last week’, or to the question ‘Of those of you who said you *do* own a bike, have you ridden it in the last week?’ A third of the children surveyed didn’t have a bike, so is the 46% an absolute number from the whole sample, or relative to the 67% of kids who owned a bike? Are we looking at bikes gathering dust, or children wishing they had a bike to ride? If we’re talking about an absolute percentage, if 46% had ridden a bike and 33% had no access to a bike, that would mean 21% of children had a bike which they hadn’t ridden during a possibly cold week in March. And yet, the headlines are screaming ‘couch potatoes’ and sports-shunning. Do the figures back this up?

6) As for 15% of children not playing with their parents, that’s an interesting texture fact – showing how parents may no longer play sports with their children – but again, what does it mean? What counts as playing a sport? Kicking a ball? Throwing a frisbee? Does a bike ride or a swim count as playing a sport? More importantly, what were the children told to consider, what was in the question? Again, this was one of the details I was able to have confirmed, and the question read:

When was the last time you played sport with your parents/guardians?

15% of children said ‘Never’ to this. Given the lack of clarification of what sports can be said to have been played, it’s not beyond imagination to think that a child could honestly say no to this, even if they’ve gone on bike rides, gone swimming and even gone for a run with their parents. In short, this question, to back up the need for the British Triathlon federation to get children into triathlon, may be too weakly-worded to even identify parents who run triathlons with their children.

7) This one is perhaps the most significant of all (which makes me wonder why I didn’t lead with it. Perhaps I like to reward persistent readers): where was the sampling taken? This, again, I can answer – when I was sent the full press release by the PR agency, it included a reference to who ran the survey… and it was indeed OnePoll, my favourite online market researchers. OnePoll have a children’s wing called YoungPoll (no sniggering at the name, please), where market research can be conducted on what children of current OnePoll members think about things. Quite how YoungPoll managed to canvas the opinions of 150 6 year olds in the time their survey was open, I’m not sure – but I’ve no reason to suspect they didn’t do exactly that, and even still the survey doesn’t require anything so murky to show potential flaws.

Now, I’m not going to give OnePoll a needless hard time here (you can see me do that elsewhere on this site), but it’s worth considering – if you want to find out how many children favour computers and technology over sports, is an internet poll the best way to go about this? There’s a potential self-selection bias at play, with the kids most likely to take the survey (or to be sat with their parents taking the survey) being the ones least likely to be out riding bikes, running and swimming.

Put another way, if this survey was conducted not online, but by a field researcher in a busy park or leisure centre, how different would the results be? Context is key.

8 and the rest) All of the above doesn’t even take into account the issue of having children – some of them very young children – take part in the survey, susceptible as other survey takers are to poorly-worded questions, unclear instructions, the urge to rush through answers quickly and all of the other kinds of factors I’ve touched on in the past. There is of course the possibility that this wasn’t a survey of children, but of their parents (or at least a mix of the two, with parental instruction to fill in the poll), which itself introduces all manner of potential biases too.

Now, I just want to reiterate, in case it wasn’t clear above – I’m not saying that any of this survey data is spurious, false, massaged, manipulated or anything of the sort. I actually think a lot of it, if not all of it, is probably on the level, with the caveat that it’s not as shocking or staggering as it might appear (reversing the spin of the stats, for example, actually shows that the overwhelming vast majority of kids can both swim and ride a bike). But I think it makes for a good example about how a story can make a lot of headlines because it has the feel of newsiness, when without the context to make sense of what’s being said, the details we’re given are barely above the level of meaningless. It’s the classic example of the three blind men and the elephant, each man able to feel one or two details each, no man having the whole picture to put together and make sense of.

Until these surveys start coming with the option to download the source data and appraise it ourselves – outside of the spins and rhetoric of PR machines – we’ll never truly be able to reliably tell if we really do have an elephant on our hands.