Ben Goldacre, The Guardian, Saturday 31 July 2010

It’s the near misses that really make you want to shoot your own face off. This week the Centre for Policy Studies has published a pamphlet on education which has been covered by the Mirror, the Mail, the BBC, the Telegraph, the Express, the Guardian, and more. Boris Johnson endorses it.

The report examines why one third of children have reading difficulties at the age of 11, and concludes it is because of a lack of discipline, and the absence of a teaching system called “synthetic phonics”. The report contains lots of anecdotes, but barely mentions the evidence.

In 2006 the government published a systematic review and meta-analysis of all the trials ever to look at phonics, which you can read in full online. Skip the rest of this boring paragraph. There were 14 trials in total looking at reading accuracy as their outcome, and collectively they found some evidence that phonics are a little better. Then there were 4 trials looking at comprehension, which found only weak evidence of benefit. Finally there were 3 trials on spelling, which collectively found no benefit for phonics. All of these trials were tiny, and when I say tiny, I mean they had between 12 and 121 children, mostly at the lower end of that range. Only one trial was from the UK.

Many teachers feel the evidence is not compelling, and don’t like phonics. To be fair, there really isn’t enough evidence to say phonics definitely works. The pamphlet recognises this. So how do we move forward? Should we run a large, well-conducted randomised trial?

No. The think tank have it all worked out, and so does Boris. Their innovative solution is taken seriously by every newspaper in the country. “It is time to end this culture war,” says Boris in the Telegraph: “to try to settle once and for all, in the minds of the teachers, whether synthetic phonics is the complete answer or not… It is surely time for the Government to organise a competition, a shoot-out between the two methods, to see which is the most effective for children of all abilities.” Both expand on this idea. Read for yourself. They don’t mean a trial. They really do want a competition.

By now you don’t need me to tell you how dumb this suggestion is, but in case anyone in power is reading: there is no room for debate here, a “competition” between schools who’ve chosen one or other method is definitely and unambiguously flawed by design. We run randomised trials, where the schools are randomly assigned to one method of teaching or another, for one very simple reason: we want to make sure that the two groups of schools – the ones doing the phonics, and the ones using the other methods – are as similar as possible for all other factors.

If we don’t randomise, “using phonics” might not be the only difference between the two groups of schools. Maybe the schools using the strict phonics systems tend also to be run – and attended – by hardworking disciplined nerds like me: if this is the case, then those schools might do better on literacy tests because of the nerdiness, rather than because of the phonics.

Why have large, robust, randomised trials not already been done? Because people like Boris don’t demand them; because teachers often believe – as doctors once did – that their expertise and intuition make such tests irrelevant and undesirable; and lastly, because many academics in the field of education inexplicably resist them.

This is a relatively new tragedy. In education, as in medicine, there is potential to do enormous good, but also incalculable enduring harm through failure: and recognising that, some of the earliest examples of randomised trials are from education. Many of these predate the 1948 MRC trial of streptomycin which is widely (and incorrectly) regarded as the first proper randomised trial. In 1928, Remmers took the worst 200 students of one freshman year and randomised them to receive either remedial teaching, or teaching as usual, and measured the difference in outcomes at the end of their course. In 1931 Walters did a randomised trial to see if counselling improves student performance. In 1933 Remmers was at it again, running a randomised trial to see if having exams at the end of the first term on a course improved a pupil’s outcome in final exams. Education researchers helped to pioneer randomised trials, a lifetime ago, but then abandoned them.

We expend a vast amount of money and effort on assessing children, without much evidence that this does them any good at all; but we make no attempt to cheaply and systematically assess the teaching profession’s various education methods, despite knowing for an absolute fact that this would bring incalculable benefits for every generation to follow. Instead we have Boris and some think tank wittering on about a “competition”: and everyone takes them seriously.