For decades the US mogul has held sway over British media and political life – but last week all that seemed to change

Shortly before nine o'clock on a Saturday evening last month an elderly man wearing a woollen jumper and slacks escorted a flame-haired woman to the back of a dining room in a Cotswolds pub. The sun was emerging after a day of rain and the jolly mood in the Oxfordshire gastropub was shared by the couple. Laughing, they settled side by side behind a stripped pine table and examined their menus.

Fellow diners scrutinising the couple attentively could have been forgiven for mistaking them for father and daughter, such was their age gap and the way they seemed to be extremely comfortable in each other's company. Whatever their relationship, clearly they were close. At one stage the woman could be seen wiping fluff off her companion's jumper.

They were still at their table, chatting casually to locals, two hours later. If they had pressing matters on their minds, they did not betray them. Only the chauffeur-driven car waiting outside the honey-stoned pub might have given a clue that they were a little out of the ordinary.

That Rupert Murdoch had chosen to spend a rare evening in the UK outside London with Rebekah Brooks, chief executive of his News International UK subsidiary, says much about the relationship between the two.

While many of their friends and colleagues, including Brooks's racehorse-training husband, Charlie, were attending George Osborne's 40th birthday party, Murdoch had chosen to spend his evening with his most loyal lieutenant, who lives close to the Kingham Plough pub, near Chipping Norton. Murdoch, who can expect presidents and prime ministers to fly all the way round the world to court him, was dropping in on his employee. The mountain was coming to Muhammad.

Although, only two days earlier, Brooks had been at Murdoch's annual summer party in London, where she had rubbed shoulders with David Cameron and the Labour leader Ed Miliband, the two would still have had much to talk about.

That party was notable for the fact that several Tory ministers, including culture secretary Jeremy Hunt, had opted not to attend, concerned about being seen to be too close to Murdoch at a time when his holding company, News Corp, was seeking a full takeover of satellite broadcaster BSkyB, a deal that rival media companies warned would cripple competition.

The putative takeover was framed by the backdrop of never-ending allegations of phone hacking at Murdoch's News of the World newspaper, which had given the media mogul's enemies plenty of ammunition to use against his BSkyB bid. How could the government endorse such a deal when one of the jewels in the crown of the Murdoch empire had been engaged in such criminality, critics asked. How could Brooks apparently have not known what was going on?

The same questions were repeated vociferously last week as evidence emerged that the phone of murdered teenager Milly Dowler had been hacked, as well as those belonging to the families of the 7/7 victims.

But Murdoch would not give his critics what they wanted: Brooks's head. For a man often labelled ruthless, it was an extraordinary defence of an employee. It was also costly. News Corp's share price dropped as analysts warned the Sky deal might be delayed.

The saga was spiralling out of control, threatening not only the Sky deal but also long-term damage to Murdoch's US interests such as Fox News and the Wall Street Journal. According to one insider, the crisis has dismayed Prince Alwaleed bin Talal whose Saudi-based Kingdom Holdings sovereign fund owns 7% of News Corp.

In a belated attempt to show how seriously it was taking the allegations, News Corp revealed that Brooks has been replaced as the head of a team investigating the phone hacking. Instead, two experienced lawyers, Joel Klein and Viet Dinh, who both sit on News Corp's board in New York, will lead the inquiry.

But it was not nearly enough. Murdoch, who was attending a conference of media bigwigs in Sun Valley, Idaho, found himself surrounded by reporters last Thursday, baying for answers. Flanked by his wife, Wendi, the ageing mogul cut a diminished figure, battling through the throng and belligerently saying he had nothing to add to a statement he made earlier in the week.

With shareholders and politicians vying to express their fury, it was left to Murdoch's son, James, News Corp's chief operating officer, to deliver the coup de grâce.

But, astonishingly, it was not to be Brooks's head on a plate. Instead it was the newspaper she edited between 2000 and 2003. The News of the World, Britain's bestselling Sunday paper, was to be axed after 168 years, Murdoch Junior revealed in an email sent to all News International staff. A fleeting visit from Brooks to the paper's newsroom, in which – soft-voiced, dry-eyed and rambling – she spoke of her affection for the paper, confirmed its demise to the few shell-shocked staff who were there to hear her.

As a damage limitation exercise, it was as brutal as it was unprecedented. But in sacrificing its massively profitable Sunday title, the Murdoch empire has triggered more questions than answers. Questions that will now dismantle what became an unholy alliance of politics, press and police.

Talk to former News of the World journalists and ask where it all went wrong and they are likely to start with Phil Hall. The combative hack, who now runs his own PR company, started his career on the Dagenham Post and became the News of the World editor in 1995. Hall inherited a paper with a circulation above four million that enjoyed a formidable reputation as a gutsy breaker of big stories. Some were famously salacious, but many involved exposés of the great and the not-so-good, big league criminals, dodgy politicians and corrupt officials.

"It was a proper paper 20 years ago," one former employee told the Observer. "We turned over drug dealers, immigration rackets, things like that. Really good, hard-hitting stories. It also made people laugh; there was lots of fun stuff in it. Sure, there was a touch of spin to it all, but the stories were genuine. We were not saints. We bent things, but it was only to get the guys who deserved to be got."

Part of the paper's success lay in the near symbiotic relationship it enjoyed with the police, the two institutions swapping tip-offs and working together on major stories that ensured a win-win for all involved: the cops got the glory; the paper the headline.

But after Hall came in things went in a different direction. Journalists were under increasing pressure to bring in stories. "The focus became celebrity and then all the other papers followed and so it became even more competitive," the former hack said.

Andy Coulson, who took over as editor in 2003, was cut from the same cloth. The man who would go on to become Cameron's spin doctor, and was arrested on Friday in relation to allegations of phone hacking and corruption, appeared to be a firm believer in the macho politics of the newsroom. A 2008 industrial tribunal found he had presided over a culture of bullying at the paper that forced one his reporters to go on long-term sick leave because of stress-related depression.

Coulson had cut his teeth on the Sun's Bizarre column, another high-octane environment. "People were having nervous breakdowns left, right and centre," recalls one former employee. "There were people crying in the toilets. Every day you put your body on the line."

Little changed when Coulson arrived at the News of the World. "Everyone felt that pressure from the executives down," said one News International employee. "Conference could be incredibly tense sometimes and maybe that pushed some people to do stupid things, but it was never overt. It was never something that people talked about it. If it was happening, and I suppose it clearly was, then people were going off to do it somewhere on their own. Andy was a really good editor and wanted good stories. He was passionate. It was tough."

Some of the staff may have felt uncomfortable, but the culture reaped dividends with the News of the World bringing in scoop after scoop that left rivals trailing in its wake well into the new millennium, when Brooks took over, editing the paper for three years before moving to the Sun.

Even if, in common with other papers, its circulation was declining, the sensational stories ensured about 7.5 million people continued to read the paper, of whom 2.7 million were the wealthy ABC1s beloved of advertisers. The News of the World was a cash cow for Murdoch, who used its profits to help shore up his other newspaper interests such as the Times and the Sunday Times, which gave him huge political leverage.

What has now become clear is that the provenance of a large number of those stories can be traced to private investigators employed by News International, several on six-figure contracts.

At the outset, in the 1980s, much of their work – such as obtaining ex-directory numbers or helping find addresses – was relatively routine. Sometimes it involved covert surveillance, even though it was not always for reasons that could be justified in the public interest. An outside agency was employed to establish that Freddie Mercury had HIV. One former journalist told how the bar belonging to the brother of a television personality was bugged. "Half the dressing rooms on [the television soap] Eldorado were also done," he said.

But the arrival of the mobile phone added a new dimension. "It used to be much easier to listen to live phone calls when it was the old analogue cell system," one former journalist said. "In the early 1990s there used to be an advert in the Exchange and Mart from a mobile shop in Bridgend which offered for sale an old Motorola carphone-type phone which had been doctored with a serial cable that could be connected to your PC. With the software provided you could use it as a live scanner showing people's numbers and listen in to calls via the PC."

Soon journalists across Fleet Street were well versed in how to listen in to the new phones and to access their voicemails. "It became more of a question of journalists listening in to other journalists' phones from rival papers," the ex-journalist said. "One journalist would deliberately leave false messages to throw people off the track of where he was and what he was doing."

Some private detectives on contract to the paper were like Glenn Mulcaire, the former footballer at the centre of the hacking scandal and a newcomer to Fleet Street. "Working for the News of the World was never easy," Mulcaire said last week. "There was relentless pressure. There was a constant demand for results. I knew what we did pushed the limits ethically, but at the time, I didn't understand that I had broken the law."

Many others were like Sid Fillery, a former member of Scotland Yard's flying squad, who worked for a private detective firm, Southern Investigations, run by his friend Jonathan Rees. The two men were accused of being involved in the unsolved murder of Rees's business partner, Daniel Morgan, but walked free after the case against them collapsed earlier this year, with the police accused of misconduct by the judge.

It is this type of complicated relationship between the police, the papers and private investigators that is likely to yield further scandal as the three sides turn on each other.

Fillery, who now runs a pub in Great Yarmouth, Norfolk, confirmed to the Observer that the agency had worked with the News of the World on a string of legitimate stories while he was in the Met. But, in a development that promises to throw more fuel on the fire, he said he intends to sue his former force. A spokesman for his solicitors, Pannone, said: "We can confirm that a partner at the firm is advising Mr Fillery on an action against the Metropolitan police for malicious prosecution."

The Met, meanwhile, is scouring all the evidence it has accumulated on Rees to establish if his firm was also involved in carrying out illegal activities on behalf of newspapers. There are said to be at least 11,000 pages of material relating to Rees in the Met's possession, none of which has yet been disclosed and some of which is thought to relate to key public figures who so far have been mentioned only on the periphery of the scandal.

Significantly, while it is confirmed that Rees was paid by the News of the World, the Observer understands other newspaper groups used his services far more extensively.

The names of other investigation agencies are likely to emerge soon as Operation Weeting, the Met's investigation into phone hacking, continues. "There were lots of other agencies working for the papers; I know of at least three more," one private investigator said.

So far the arrests have been confined to reporters and editors, but how did the investigators obtain the mobile phone numbers to hack into in the first place? One obvious line of inquiry is the illegal accessing of the police national computer, suggesting corrupt officers were involved. The paper has already confirmed that several Met officers were paid for information.

But there will be others outside the force. "I should imagine there are some ex-BT engineers that have done extremely well over the years performing dark arts via third parties," said one former News of the World employee.

A News International insider said that claims an estimated 4,000 phones may have been targeted could tell only part of the story. There are suggestions that the paper was interested in as many as 80,000 phone numbers over the past decade. How many were hacked or bugged is a subject for the police investigation, but by the mid-1990s it appears hacking had become endemic and no one was considered out of bounds. From the families of 7/7 victims to Milly Dowler, all were targets. John Cooper, a barrister who represents the families of soldiers killed in the Nimrod disaster in Afghanistan and the RAF Hercules explosion in Iraq, as well as those who died at Deepcut barracks, confirmed on Saturday night that his clients were concerned that they may have been the victims of telephone hacking.

Even the nearly dead were apparently fair game. In the winter of 2004, when his most famous client, George Best, was dying of liver failure, agent Phil Hughes could not understand how the press appeared to be outside the right hospitals at the right time.

"Somehow the News of the World always seemed to understand who was visiting and would always have photographers there," said Gerald Shamash, Hughes's solicitor, who has asked the Met to hand over any information it has relating to his client.

"Phil is convinced his phone was substantively hacked by the News of the World. The situation became very difficult, particularly in the latter months of George's life. It was very upsetting for both of them."

As the story switched last week from hacked celebrities to vulnerable members of the public, the mood noticeably shifted. In the City, BSkyB's shares took a pounding as Ofcom, the media regulator, said it would consider whether News Corporation would make a "fit and proper" owner of BSkyB. By the end of the week the shares were down nearly 12%, wiping £1.8bn off BSkyB's market value as hedge funds bet the deal would be bogged down for months to come.

The fit and proper person test applies to any owner of a TV station in the UK. The regulator indicated it would invoke the test only if a director of BSkyB were to be charged with criminal offences, such as phone hacking.

But other legal concerns are brewing. There is speculation that illegal acts by company executives in London could potentially be prosecuted in America under the Foreign Corrupt Practices Act which is aimed at stopping US firms from engaging in bribery abroad.

At the same time, the idea has been floated that News of the World journalists, or those working at News Corp organisations in the US, might have broken the law in pursuit of stories across the Atlantic. The US has extremely strict laws on phone hacking and many ambitious prosecutors might like to make a name for themselves by pursuing such a case.

In the face of massive public opprobrium and a City backlash, James Murdoch's decision to kill off the title was portrayed as a kneejerk reaction, an emergency amputation to keep the News International patient alive. But this may not be true. One well-placed source has suggested Murdoch has had a team working on plans to replace the News of the World with a Sunday Sun for at least three months. This belief is shared by former journalists on the paper. "What happened on Thursday was a cynical exercise to save Murdoch money, sack staff and turn the Sun into a seven-day operation," said one. "Thirty years ago this would have been a trade union issue, but Murdoch did for that."

Analysts were quick to pronounce that closing the News of the World was a small price for Murdoch to pay. True, the paper is highly profitable, making an estimated £12m of profit in 2010 and generating almost £50m in advertising revenue. But Sky, in which News Corp owns a 39% stake, is forecast to make more than £1bn profit in 2011-12.

On Wall Street, Richard Greenfield of US broker BTIG said Murdoch's other media interests in cable television – Fox News and his numerous other operations – were far more valuable in the eyes of investors than print.

Greenfield spoke for his fellow analysts when he said: "Many of us believe newspapers are a sunset industry and wouldn't give a damn if Murdoch decided to get rid of them."

Murdoch's audacious overnight transfer of his newspapers to Wapping, east London, in 1986 proved he hated the trade unions, but what he likes is more difficult to pinpoint. In an interview with the Village Voice newspaper in 1976, seven years after he bought the News of the World, he gave a rare insight into his psychology. He painted himself as an outsider, someone who rubbed up against the grain.

"I just wasn't prepared to join the system," he said. "Maybe I just have an inferiority complex about being an Australian … you join the old school-tie system and you're going to be dragged into the so-called social establishment somehow. I never was."

His status as an outsider was confirmed shortly after he acquired the News of the World when it published the diaries of Christine Keeler at a time the shamed minister, John Profumo, was trying to put the scandal behind him. However, it was Murdoch's purchase of the Times, waved through by Margaret Thatcher in 1981, and the paper's subsequent move to Wapping that saw him become a member of the establishment he professed to loathe.

Murdoch and Thatcher were ideological soulmates who espoused free markets, loathed Europe and were impatient to dismantle the UK's creaking old institutions. For once, Murdoch seemed to have genuine affection for a politician, usually seeing them as useful allies in his quest to expand his interests.

This political pragmatism plays to Murdoch's advantage, allowing him to back winners – and oppose losers. It was only in 1992, when John Major won a surprise election victory over Neil Kinnock's Labour party, that the full extent of Murdoch's influence became evident. Kinnock had looked on course for victory but the Murdoch press led a strident campaign against him in the final days.

On the morning of election day the Sun front-page requested that, "If Kinnock wins today will the last person to leave Britain please turn out the lights". As he licked his wounds amid the wreckage of a fourth consecutive general election defeat for Labour, Kinnock blamed the media and the Murdoch stable in particular for turning the tide against him. "It's The Sun Wot Won It" ran the paper's triumphant headline.

From that moment, Labour's modernisers – Tony Blair, Peter Mandelson, Gordon Brown, Alastair Campbell and Jonathan Powell – knew that if the party was to break the Tory stranglehold on power there no more important task than to get Murdoch and his papers onside.

Lance Price, a journalist and ex-spin doctor who worked at No 10 as Campbell's deputy, recounts how Blair and Campbell took to heart the advice of the Australian prime minister, Paul Keating, on how to deal with Murdoch.

"He's a big bad bastard and the only way you can deal with him is to make sure he thinks you can be a big bad bastard too," he said. "You can do deals with him, without ever saying a deal is done. But the only thing he cares about is his business and the only language he respects is strength."

Throughout his years in power, Blair had regular secret meetings with Murdoch, many abroad, and was in regular telephone contact. Price has gone as far as to claim that Murdoch "seemed like the 24th member of the cabinet".

Blair insisted no record was ever kept of the meetings or calls, so they were totally deniable. Cherie Blair has said that her husband's decision to go to war in Iraq in 2003 was a "close call". So it was – and there is evidence that the final decision was taken only after Murdoch's encouragement was received and his blessing given. Blair talked to the media tycoon three times on the telephone in the 10 days before the US-led invasion. Details obtained under freedom of information show Blair called Murdoch on 11 March, 13 March and 19 March 2003. British and US troops began the invasion on 20 March, with the Times and Sun voicing total support.

The Murdoch penetration into the heart of political life has accelerated under Cameron. His links to the Murdoch empire are arguably even closer than those of Blair or Gordon Brown, whose wife, Sarah, helped to arrange Brooks's 40th birthday party.

The contact between the Tory leader and the likes of Michael Gove, the education secretary and an ex-Times journalist, are not merely professional but also social. They mix in the Oxfordshire political and media set. Cameron, who has been a guest at Brooks's Cotswolds home, made his own visit to see Murdoch in August 2008 on his yacht off the coast of Greece.

But after last week's momentous events some are questioning whether the umbilical cord between Murdoch and Britain's politicians has been snapped. Some commentators wonder whether, in an era of declining sales, the hegemony of the press, and in particular that of Murdoch, has been overstated. The rise of new media is allowing politicians to convey their message without needing newspapers as an intermediary. Advertisers are shifting their spending from conventional media brands to social networking sites.

MPs, who last year were accused by Lib Dem deputy leader Simon Hughes of being "too scared" of Murdoch's News International to testify in court that their phones had been hacked, are lining up to denounce the mogul. "We are in a totally new world now," said one shadow minister. "This is unbelievable. The Murdoch empire, in a matter of hours, has gone from being one which politicians wanted to do everything they could to please, to one they were desperate to disown and condemn. Murdoch has turned from asset to liability."

The replacement of the Press Complaints Commission with an independent regulator, after the watchdog was roundly criticised for failing to get to grips with the scandal, will further curtail the power of newspapers.

Two official inquiries, one into phone hacking, the other, with a wider remit into press ethics, promise uncomfortable headlines for Fleet Street over the coming months. So too does Scotland Yard's continuing investigation, the results of which will extend far beyond the News of the World and phone hacking to other newspapers and criminal acts like bugging and email interception.

Brooks herself hinted there was much more bad news to come, telling staff they would only understand why the plug had to be pulled on their newspaper a year down the line – presumably when criminal investigations have concluded.

Last Thursday evening, stunned News of the World staff made their way to the Cape bar in Wapping where they watched constant updates of their demise flash up on large television screens. It must have been a strange feeling. Used to making the news, they were the news.

A ripple of applause from the table occupied by staff on the paper's Fabulous magazine greeted an announcement on Sky News that subeditors at the Sun had briefly walked off the job in protest at their sister paper's closure. Most of the anger was saved for a solitary figure – Brooks. Picture editors vied with subs and young reporters to say the same thing: they had been sold down the river by the Murdoch family to save her skin."There are young people with families," one said. "What are they going to do?"

Their mood is unlikely to be helped by the disclosure, presumably made by a disgruntled, recently unemployed member of staff, that Brooks regularly enjoys the services of a helicopter to fly her from Battersea heliport to her Cotswolds home. Her use of a private jet for a breakfast meeting in Venice is also the subject of discussion by Wapping veterans.

"This is about what happened under the old regime," volunteered a senior reporter gesturing to the pub's television screens. "Look at most of these people. They weren't even around when all this happened. Colin Myler [the paper's editor] might have his faults but he was trying to turn it round. We've all been sacrificed to save Rebekah Brooks."

Their anger raises an important question. How will reporters and editors of other Murdoch titles such as the Sun and the Times feel about continuing to work under Brooks, especially after Cameron in effect called on her to stand down, saying: "It's been reported that she had offered her resignation in this situation, and I would have taken it." His comment again threw into question Murdoch's increasingly quixotic desire to protect Brooks. As the seasoned media commentator Raymond Snoddy observed on the MediaTel Newsline Bulletin: "Her famed political access will be no more. You can hear the doors already slamming in her face."

But her weakened stature will mean little to the 250 staff on the paper now out of work at a time when none of its rivals is hiring.

In an email to staff yesterday, Myler said: "You have made enormous sacrifices for this company and I want you to know that your brilliant, creative talents have been the real foundation for making the News of the World the greatest newspaper in the world."

On Saturday night, as Murdoch prepared to fly in to Wapping to tackle a crisis that refuses to die, the News of the World was doubling its print run to five million, anticipating a surge in demand from readers keen to buy a piece of history. Whatever plans he has for its replacement, it was a curtain coming down. Not just for the News of the World but for all of Fleet Street.

To comment on this story or any other about phone hacking, visit our open thread from 9am