Britain's drama has penetrated the carapace of American self-preoccupation. Legendary reporter Carl Bernstein compares it to Watergate. On morning television, Hugh Grant appeals to Americans to wake up to Rupert Murdoch's pernicious influence on their own media. Business reporters track the impact on News Corp shares. Senator John Rockefeller calls for an inquiry into whether Americans' phones were hacked. If it turns out that 9/11 victims were targeted, as suggested by the campaigning MP Tom Watson in prime minister's questions, then this will no longer be just a foreign story. Only on Murdoch-owned Fox News is it as if none of this had really happened. A clip from Fox News Watch, filmed during a commercial break, shows the panellists joking about the one story they are not going to discuss. News watch indeed.

Illustration by Matt Kenyon

But what does it all mean? "A kind of British spring is under way," writes the media columnist David Carr in the New York Times. "Democracy, aided by sunlight, has broken out in Britain." Hyperbole, of course, but he has a point. I'd put it like this: the Murdoch debacle reveals a disease that has been slowly clogging up the heart of the British state for the last 30 years. This is the heart attack that warns you that you are sick, but also gives you the chance to emerge healthier than before. The root cause of this British disease has been overmighty, ruthless, out-of-control media power; its main symptom has been fear.

To talk of a British spring, by analogy with the Arab spring, is obviously poetic exaggeration. Compared to most other places in the world, Britain is a free country. In many ways, it is a better one now than it was when Murdoch bought the Times in 1981. But at the apex of British public life there have been men and women walking around with small icicles of fear in their hearts; and fear is inimical to freedom.

This was a fear that dared not speak its name; a self-deceiving cowardice that cloaked itself in silence, euphemism and excuse. Inwardly, politicians, spin doctors, PR men, public figures and, it now emerges, even senior police officers, said to themselves: don't take on Murdoch. Never go up against the tabloids. Murdoch & Co used shameless, unscrupulous and illegal intrusions into privacy both to sell newspapers, by titillating a celebrity-hungry public with intimate details, and to secure political influence.

If the tabloids had not actually gone after you, the threat was always there. In corrupt, thuggish Russia, they call it kompromat: compromising material, ready to be used if you step too far out of line. We now know that the hacks and their hackers stopped at no one and at nothing. The royal family, families of soldiers killed in action, kidnapped children – all were targets for intrusion and exposure.

Overweening media power has also shaped British policy in important ways. Contemplating the ruins of Tony Blair's well-intentioned attempt to resolve Britain's chronic indecision about its place in the European Union, an attempt destroyed by the Eurosceptic press, I once concluded that Murdoch was the second most powerful man in Britain. But if the ultimate measure of relative power is "who is more afraid of whom?" then you would have to say that Murdoch was – in this narrow, hard core sense – more powerful than the last three prime ministers of Britain. They have been more frightened of him than he of them.

Consider the evidence. Blair had seen his predecessor as prime minister, John Major, and a Labour leader, Neil Kinnock, destroyed by a hostile press. He learned his lesson. He wooed those press barons for all he was worth. Only as he was about to leave office, after 10 years, did he dare to denounce the British media for behaving "like a feral beast".

This week we learned that Blair's successor as prime minister, Gordon Brown, believes his family's bank and perhaps tax records were hacked or blagged into. Brown tells us he was reduced to tears after the then editor of the Sun, Rebekah Wade, rang him to say that the paper was going to reveal that his four-year-old son Fraser had cystic fibrosis. Yet a few years later Brown still attended the wedding of said Rebekah – who is now, as Rebekah Brooks, Murdoch's right-hand woman at News International. The Morgan le Fay of British journalism was just too powerful for a prime minister seeking reelection to slight.

David Cameron out-Blaired Blair in wooing the press barons in general and Murdoch in particular. Worse, he hired the former editor of the News of the World, Andy Coulson, to be his communications director. I can not recall meeting anyone in British journalism who believed the ex-editor was as innocently unknowing as he claimed of what his reporters had been up to. But Cameron ignored all the warnings he was given.

Most shockingly, the Metropolitan police shelved an investigation that they should have pursued vigorously. They failed to tell thousands of people whose names appeared in the books of a private investigator that their phones might have been hacked. Only tenacious investigative reporting in the Guardian and the New York Times forced a reopening of the police investigation.

Perhaps the single most important thing the promised public inquiry now has to establish is why the police acted as they did. Here again, the most plausible explanation boils down to fear. The police were afraid of imperilling their cosy relationship with the Murdoch papers, which helped them in their inquiries and praised them for their crime-fighting efforts. Some police were paid by the Murdoch press. Senior officers now say that their own phones were hacked. Absent strong evidence to the contrary, the only reasonable conclusion is that the police feared being mauled rather than embraced by the feral beast. So they, too, bent the knee.

All that remains is for us to discover that a senior judge was spied upon, won over or intimidated. "Surely not!" we cry. "Not that!" But how many times before have we thought that we had reached bottom, only to hear knocking from underneath?

Yet even if there are still worse revelations to come about the past, the future looks brighter. The best of British journalism has exposed the worst. In parliament, the worms have finally turned. Party leaders and ordinary MPs are, at long last, reasserting the supremacy of elected politicians over unelected media barons. The barrier of fear has been overcome.

Out of this putrid quagmire there should emerge a whole new settlement: in the balances between politics, the media, the police and the law; in the self-regulation of the press and in the practice of journalism. The danger is that, once the initial outrage has passed, Britain will again settle for half-measures, half-implemented, as has already happened with the impulse for constitutional reform that came out of the parliamentary expenses scandal. But for now, one of the most important crises of the British political system in 30 years has produced an opportunity. I will return this autumn to a Britain that is slightly more free.

• Comments on this article will be turned on at 9am UK time

• This article was amended on 15 July 2011 to remove references to the obtaining of "medical records" - in line with the following correction, that appeared in the Guardian on 15 July 2011:

Articles in the Guardian of Tuesday 12 July incorrectly reported that the Sun newspaper had obtained information on the medical condition of Gordon Brown's son from his medical records. In fact the information came from a different source and the Guardian apologises for its error (The Brown files: How Murdoch papers targeted ex-PM's family, 12 July, page 1; When Brown decided that the Sun was out to destroy him politically, 12 July, page 2).