“You now have a shorter attention span than a goldfish,” Time reported last week in its “Neuroscience” section. The Telegraph got a little more specific about the cause: “According to scientists, the age of smartphones has left humans with such a short attention span even a goldfish can hold a thought for longer.”

The statistic being reported by myriad publications is this: in 2000, the average attention span was 12 seconds. Now, it’s just eight seconds. The widely-touted attention span of a goldfish is nine seconds. Hence, smartphones have turned the tide on human cognitive evolution and we now have less capacity to remember where our plastic castles are than goldfish do. Up next, our opposable thumbs are set to retract into our hands, and then they’ll come for our bipedalism.

Obviously, this is nonsense, but the truly astounding thing is the sheer extent of the nonsense. As a starting point, here’s the source of this “scientific” statistic: a Microsoft marketing report. The report is targeted at marketers, suggesting ways to produce marketing messages suited to tech-savvy, multitasking, easily bored demographics. Consisting of a survey, behavioural tests, and electroencephalography (EEG) tests (presumably for the cool publicity shots of electrode-studded heads), the report explores the relationship between tech-related behaviours and attention spans.

Goldfish don't do web searches

The goldfish statistic isn’t actually based on Microsoft's research. It’s included in the introduction, somewhat mysteriously, since the report actually emphasizes the opposite conclusion: “Think digital is killing attention spans? Think again.” The source, Statistic Brain, doesn’t cite studies for its “less than a goldfish” statistic, but does lead to a 2008 paper finding that after a web search, people stay on pages they open for an average of eight seconds.

It’s not exactly the most direct measure of attention span. In fact, according to University of Aberdeen attention researcher Søren K. Andersen, it’s not a measure of attention span at all. The ability most people refer to when they talk about attention spans is what researchers call “vigilance." It's a measure of how long a person can stay focused on a task, when staying focused is their goal.

It’s vital to distinguish between what people choose to do when there’s no reason for them to focus, and what they’re capable of doing when focusing is the point, Andersen told Ars in a telephone interview. People might find it pleasant to flick from one activity to another when that’s an option, but might still be perfectly capable of putting their minds to concentrating when the situation requires it.

There’s also the matter of what constitutes the most efficient behaviour in a given situation. In the middle of a web search, sitting on an irrelevant web page for longer than eight seconds probably isn’t efficient. “If your room is messy and you’re looking for socks, it’s most helpful to attend to each item briefly; focusing on one thing for a long time isn’t an adaptive strategy,” says Andersen. So web page load times—obviously—don’t tell us anything about shortening attention spans.

Missing data, confounds, and insignificance

So, the goldfish headline comes from an old statistic, isn’t a measure of attention, and isn’t really the focus of the Microsoft report. What does the report actually say, then? It summarises its own findings as showing that digital lifestyles erode attention spans, but people are better at multitasking, are doing more with shorter bursts of attention, and have “more efficient encoding to memory.”

The evidence for these claims is extraordinarily difficult to assess, because the research is not peer-reviewed, and doesn’t adhere to the standards of peer-reviewed reporting. The research methods and data analysis aren’t described in enough detail to be able to properly analyse them. Andersen points out that the methods could be entirely respectable; the research might even have been submitted to a peer-reviewed journal. But the scant evidence, as presented in the report, raises a number of questions.

Three types of attention are identified. Sustained attention—what Andersen would also refer to as vigilance—refers to “maintaining prolonged focus during repetitive activities,” and is tested with a game similar to a conventional vigilance test, asking participants to find any X that appears after an A in a group of letters. Alternating attention is tested with what Andersen identifies as a widespread task-switching assessment, showing pairs of letters and numbers like “K8” and asking participants to identify whether the letter is a vowel or consonant, or the number is odd or even, switching between the two kinds of tasks.

Finally, there’s selective attention, described as “maintaining response in the face of distracting or competing stimuli.” There’s not enough information here to identify the exact concept being described, says Andersen. Overall, however, he says the classifications of the kinds of attention are reasonable.

Based on a behavioural test and survey conducted on 2000 participants, the Microsoft report describes results showing that attention diminishes when people have a high rate of web browsing, high social media usage, early adoption of new technology, and simultaneous use of multiple screens.