Many months before the Panorama investigation into Operation Midland was broadcast in October 2015, I sat in a cafe in central London with a very senior officer who had just left the Metropolitan police. He was there as a go-between, doing a favour to ex-colleagues; an unofficial channel, trying to convey the Met’s new thinking about investigations into historic sexual abuse, in light of their failings over Jimmy Savile. I asked him about his general approach when he came across allegations like the ones being made by the man we knew then as Nick, and now as Carl Beech. The officer spoke plainly: as a principle, he told me, he tended to assume in cases like this that there was no smoke without fire.

There ought to be a general rule that a celestial alarm bell will ring every time a police officer says that. But none rang, neither in Scotland Yard nor in enough newsrooms around the country. Operation Midland continued for more than another year.

With hindsight, Midland can look like a moment of institutional madness. How was it possible that a set of allegations as fantastical, as grotesque, as Beech’s could be believed? That Ted Heath, Leon Brittan, Lord Bramall, the heads of MI5 and MI6 and other powerful men, in rooms in London, all had taken their clothes off and tortured, raped and murdered boys, every trace of their crimes covered up?

Now that the last, slender threads of that story have unravelled, the police need to understand – as do we – how they came to take a decision to believe Beech: the decision, in effect, to abandon evidence-based policing.

Under immense pressure after the truth about Savile was revealed, the police came to think that the old ways of investigating didn’t work any more. In many respects they were right; lots of them hadn’t worked. Genuine victims had been deterred from coming forward, and crimes which might have been solved had gone unpunished. But the right diagnosis was followed by a spectacularly wrong prescription. Determined to do nothing to undermine their witness, Operation Midland closed its eyes to facts that might turn out to be inconvenient.

There were places to start an inquiry that might have seemed obvious: the alleged murder of a boy – a schoolfriend of Beech’s named Scott – which Beech claimed had taken place in plain sight on a public road at a busy time; or talking to Beech’s wife of many years. For months, the police failed to investigate either. The Panorama team was astonished to find we were ahead of the police in contacting both Scott and Dawn Beech. There was the school attendance record that categorically undermined Carl Beech’s claim to have been repeatedly absent from school while he was being abused. It was Northumbria police’s investigation into Operation Midland which uncovered that, not Midland itself.

Facebook Twitter Pinterest ‘Under immense pressure after the truth about Savile was revealed, the police came to think that the old ways of investigating didn’t work any more.’ Photograph: PA

In the case of Scott and the school records, no deep intrusion into Carl Beech’s life would have been needed to corroborate or disprove his story. There would have been no serious risk of victim shaming. But by now the police were operating from a different playbook, one that relied on publicity rather than diligence to encourage witnesses to come forward. It had worked in Operation Yewtree, the investigation into high-profile sexual offences by Savile and others. In Operation Midland it should quickly have become obvious it wasn’t working. No credible witnessescame forward to support Beech’s story, but the police persisted.

Revealed: how Carl Beech, the serial child abuse accuser, became the accused Read more

What to make now of the way Midland was conducted? Fundamentally this: if you think the police have a steady compass, they don’t. They’re twitchy, obsessively concerned with public opinion, easily moved by political pressure. In overcompensating for their failings over Savile they drove a steamroller over the rights of innocent men.

If you worry about one consequence of Operation Midland, it should be this: it put the public in danger. Carl Beech, a paedophile and a risk to children, remained at large for years longer than necessary because the police failed to investigate him fully.

• Ceri Thomas is editor at news startup Tortoise and a former editor of Panorama and the Today programme