A family meal on a Sunday night is a traditional event. Yet when Rupert Murdoch gathered his clan together earlier this week it was to discuss a crisis and the only thing on the menu was how to save the bid for BSkyB, the fast-growing satellite business and jewel in the crown of the Murdoch empire.

Just a few days later, that effort has proved futile.

By the time David Cameron stood up at midday on Wednesday to say that the Murdochs should "stop talking about mergers when there is such a mess to sort out", the family had already decided to ditch its offer for the 61% of BSkyB it did not already own.

Yet instead of stemming criticism of the company and its management of the phone hacking scandal, the failure of the bid has instead raised serious questions about the ageing media mogul and his desire to hand control of one of the world's biggest media companies to his youngest son, James.

"These guys are on the run," said Michael Wolff, biographer of the Murdochs. "They had no option but to drop the bid. But now the real issue is how to avoid further humiliation. They are in retreat with no real business in the UK anymore, just a set of disintegrating assets."

For the first time in years, questions about the younger Murdoch's position as the chairman of BSkyB are being asked by analysts and shareholders of the £12bn satellite business.

Even close observers of the Murdochs and News Corporation were surprised at the speed of this week's events. How could a bid that was so prized and that had seemed so inevitable a few days earlier have been dumped in such ignominious circumstances?

It was a little over three weeks ago that David Cameron and Ed Miliband mingled with senior News Corp executives and industrial leaders in a grand party at the Orangery in Kensington, west London. At the time the only barrier to a successful takeover seemed to be the price likely to be offered by News Corp.

As champagne and oysters were served, one attendee said it now seems like "an orgy at the end of the Roman empire". As thunder and lightning interrupted the event, few of those in attendance could have known what a storm lay ahead of them.

The story of the past year will make a great film of course – Hugh Grant, the de facto spokesman for the Hacked Off pressure group, could even play himself. Two key scenes have played out over the past 10 days – events that led the Murdochs to decide that too much was at stake in pursuing a bid that had seemed theirs for the taking.

The first deeply affected public opinion and therefore galvanised parliament and even a government that had seemed so sanguine about the bid. The second galvanised the Murdochs themselves, who realised that not only was the takeover at risk but also their existing stake in the highly lucrative satellite business.

The endgame began with the Guardian story on 4 July about the News of the World hacking into the phone of the murdered teenager Milly Dowler. Some messages may have been deleted, to make room for more messages, misleading police and her family.

Police later said other victims of crime such as the two girls killed in Soham could also have been targeted by phone hackers at the paper. The revelations and the link to Rebekah Brooks, the chief executive of News International who was editor of the Sunday tabloid at the time of Dowler's death, shocked millions of people who had previously been unaware of the hacking scandal.

After months in which he and his senior ministers had referred to the letter of the law when asked about BSkyB, the prime minister finally snapped that he was "revolted" by allegations. News International stressed its co-operation with police – handing over evidence of payments to police and removing Brooks from the chain of command for the clean-up operation – but this failed to stem the stream of stories about hacking soldiers and others.

James Murdoch's apology and decision to close the UK's biggest-selling paper, the News of the World, as well as the departure of several key executives, including legal managers John Chapman and Tom Crone and editor Colin Myler, did little to stop the criticism from parliament and public opinion. It was simply considered too little, too late.

"It's a bit like discovering gangrene," said someone close to the situation. "If you wait too long to cut off an arm you end up having to cut a bit more off and then a bit more. Wait too long and you could die."

By the time the last edition of the tabloid was printed on Sunday – trumpeting itself as "The world's greatest newspaper" – the coup de grace to Murdoch's ambitions had already been delivered.

On Friday Ofcom announced a review of whether News Corp and its beleaguered executives would pass the "fit and proper" person test which governs the ownership of broadcast licences. Ed Richards, the Ofcom head who has proved himself no friend to the Murdochs in the past, said the regulator would "consider any relevant conduct of those who manage and control such a licence" once the criminal investigation into the News of the World had been completed.

The impact of this statement cannot be overstated. Before then, the regulatory authorities had only been asked to consider whether the owner of two of the country's biggest selling tabloids as well as the Times and Sunday Times should be allowed to take over its biggest satellite business on the grounds of media diversity, or plurality. By raising questions about whether company directors were "fit and proper" the new test could endanger Murdoch's existing 39% stake, let alone the takeover.

The Murdochs would be furious at any suggestion that they could fail a test passed by Richard Desmond in buying Channel Five. Yet the very fact that it was raised by Ofcom saw shares in Sky fall even further.

By the time details of further intrusions into the private affairs of former prime minister Gordon Brown emerged on Monday, the writing was on the wall.

The timing of the decision to drop the BSkyB bid, coming so soon after the prime ministerial statement and special debate on an issue that united all three main political parties, made it appear to be a straightforward victory for the people and their elected representatives against the most powerful media baron in the land. "It stopped being about the process and became about politics," said someone close to the deal.

The political angle to this has seen a belated but effective intervention by the Labour leader, Ed Miliband. He can legitimately claim some kudos for helping to force the country's most powerful media mogul to backtrack, but it has not always been as straightforward inside his office. In a memo written in January Tom Baldwin, the director of strategy, had requested that "any front-bench spokespeople use the following line when questioned on phone hacking. BSkyB bid and phone-tapping … these issues should not be linked. One is a competition issue, the other an allegation of criminal activity."

It went on: "Downing Street says that Cameron's dinners with Murdoch will not affect Hunt's judgment. We have to take them at their word."

Yet as the evidence of a cover-up started to accumulate and a slow trickle of arrests started, Miliband wanted to intervene. In late April, ahead of the local elections on May 6, he conducted an interview with the Guardian, and his aides indicated he would be willing to say something about the phone-hacking drama being pursued by the newspaper. In the back of a campaign car, he said: "I think there does need to be a review after the police inquiries have been completed and any criminal cases that flow from it. I think it is in the interests of protecting the reputation of the British press that these matters should not simply be left to rest, and lessons have to be learned."

Privately some of Miliband's advisers were arguing that he should be willing to confront Murdoch as it fitted with his wider responsibility agenda, the leitmotif demand that there had to be responsibility at the top as well as the bottom. But caution remained rife in his office.

In a sign of the pull Murdoch held, Miliband agreed at short notice in June to speak at a meeting of Murdoch's chief executive conference, filling in for the Chancellor, George Osborne, who had cancelled at short notice.

Once the Dowler revelations emerged, Tom Watson, the dogged campaigner against News International and a political ally of Miliband, vented his frustration at the silence of the political leaders at Westminster.

"Surely now we should hear from David Cameron and Ed Miliband," he said. "It's utterly disgraceful that they've let this scandal run on for as long as it has. No more cowardice – we want action."

At his regular Tuesday morning meeting Miliband asked his aides how to react. The meeting is now known in Labour circles as the "sod it" meeting. Even then some of his aides counselled caution, arguing he might alienate News International irretrievably. Miliband just found the whole story so shocking, he said he could not hold back. He decided to cross a potentially dangerous line by calling on Rebekah Brooks to consider her position. He also explicitly called for a public inquiry, rather than a simple review. Once he had crossed this line, there was no turning back.

The other surprise element was the willingness of the Speaker, John Bercow, to grant an emergency debate last Wednesday to the backbencher Chris Bryant. By the end of the debate, the scale of the cover-up by News International had become as important as the scale of the phone hacking. As well as Cameron's disgust, it was also becoming clear support for the Murdoch organisation was evaporating on Tory benches.

Miliband then decided he could use yesterday's Opposition debate to press the issue of the BSkyB takeover. On Monday, through Liberal Democrat culture spokesman Don Foster, Miliband consulted the Lib Dems on the wording of the motion.

Late on Monday Miliband agreed the wording should simply state that News Corp should withdraw the bid. That won the consent of figures such as Tim Farron, the party president, and Foster. Miliband then went to the leaders of other political parties, so that by Tuesday lunchtime cross-party support was clear. Cameron knew he had no choice but to support the motion, meaning parliament was united against BSkyB. The Guardian splash headline read "Parliament versus Murdoch".

Claire Enders from Enders Analysis welcomed the news and praised the early involvement of Vince Cable, the business secretary, who lost his responsibility for the proposed merger after being recorded as saying that he was "waging war" on the Murdochs. "He was the first person to really see the importance of this," she said.

Interestingly, it was left to Chase Carey, the extravagantly moustachioed number two to Rupert Murdoch, to announce in a statement that "it has become clear that it is too difficult to progress in this climate". He went on to stress that News Corp "remained a committed long-term shareholder in BSkyB."

The failure of the bid after a tumultuous 11 months could have left Sky's shareholders less committed to a family that had previously seemed infallible. Sky's independent directors, led by Nicholas Ferguson, are understood to be keen to show their strength to investors who are understandably concerned about events. Some rue the Murdochs' refusal to offer a higher price, which could have seen the deal done by now. "If they'd offered 800p at the beginning of all this, the deal would have been done before any of us knew about Milly Dowler's phone," sniffed one financial adviser.

As it is, James Murdoch's involvement in the scandal has led the FT's influential Lex column to write of a "Murdoch Jr discount" at Sky.

Insiders suggest all key decisions have been taken by family members in recent days, yet Wolff senses a difficult changing of the guard. "This is like a collapsing House of Cards, and I'm just not sure if Rupert is in a leadership position."

After leaving the restaurant on Sunday night, the waiting photographers caught James helping his increasingly frail 80-year-old father across the road. His failure to get to grips with the phone-hacking scandal at News International has called into question his place at the head of his father's business, and no amount of hand-holding is going to change that.