Labour's poll lead is at its shakiest since George Osborne's 2012 omnishambles budget, and the questions that have dogged Ed Miliband's leadership from day one are as loud as they have ever been. "Labour chose the wrong Miliband," the many commentators have long asserted. So let's look at a parallel universe where David Miliband did indeed defeat his brother four years ago …

"Today, the Labour movement chose with its head over its heart," declared the newspapers after his victory. David Miliband won, above all, because he seemed to have that elusive "prime ministerial" quality. But, after a short-lived honeymoon period, his leadership became racked by crisis. Labour angrily attacked a coalition inquiry into New Labour's complicity in the torture of foreign terror suspects as being "blatantly politically motivated": even some on the right questioned whether the inquiry would have been quite so resourced if Labour was not led by its former foreign secretary. Former Liberal Democrat voters disgusted with Nick Clegg's perceived betrayal began flocking to the Greens, rather than Labour.

In the name of party unity, Miliband initially appointed Ed Balls as shadow chancellor. Yet Balls's mild Keynesian tendencies collided with Miliband's determination "not to become the party of protest", as he put it. With Miliband pushing for Labour to indefinitely commit to George Osborne's spending plans, a new incarnation of the great Brown-Blair split re-emerged at the top of the party. After furious rows began to publicly surface, Balls was replaced by Alan Johnson: although regarded as one of Labour's most affable figures, he became relentlessly mocked for his lack of economic expertise.

An iron law of British politics appeared to be confirmed: that when Labour loses an election, the party descends into raucous division. Despite three years of zero growth, the Labour leadership merely quibbled over the nuances of the government's austerity drive. Labour's decision in January 2013 to abstain on the welfare uprating bill – which slashed in-work and out-of-work benefits – triggered frontbench resignations and a huge backbench rebellion. When the bedroom tax was introduced, Miliband declared that, "although we would not have made this choice, in the name of fiscal responsibility we cannot promise to reverse it."

Backbenchers and party grassroots became ever more angry. Three backbenchers resigned the party whip. Unite led a mass disaffiliation by trade unions; and, with bankers and business people unwilling to back a party riven by infighting, Labour faced bankruptcy. As the economy began a belated recovery, a resurgent Conservative party asked if the public would entrust the country's future with the opposition.

Miliband's leadership became known for its series of missed opportunities. The weak criticism of the government over the phone-hacking scandal caused bewilderment – and then yet more grassroots fury when it emerged there had been behind-the-scenes attempts to woo the Rupert Murdoch empire both before and after the revelations. On some issues Labour attempted to outflank the government from the right, savaging Ken Clarke for being "soft" on law and order, but as crime figures continued to fall, and the public becoming less concerned on this matter, this strategy backfired.

David Miliband with a banana on his way to conference. Photograph: Jeremy Selwyn / Evening Standard/Rex Features

The personal attacks on the Labour leader were cruel: he was portrayed as an arrogant, aloof geek. Taking unflattering pictures of Miliband goofily holding various fruits became a national sport. Polls suggested he was seen as more "out of touch with everyday concerns" than David Cameron. His wonky language was ruthlessly mocked by Cameron at PMQs.

Labour's support for the disastrous US-British bombing of Syria brought back bad memories of Blair and Iraq. And as well as triggering regional conflict, the intervention appeared to shore up Syrian support for President Assad. The west was now portrayed as the new allies of al-Qaida. As the tide turned in favour of the dictator, every time a rebel-held town fell to the regime, the misjudged western bombing campaign was blamed.

On the eve of the last Labour conference before the 2015 general election, the Chilcot inquiry into the Iraq war was finally published, and the conclusions were damning for Blair and New Labour. Miliband's protestations that he was not then a member of Blair's government fall on deaf ears: his own vote for the war, and his status as the New Labour continuity candidate, was relentlessly scrutinised. His response was another of the unconvincing statements that had become the hallmark of his leadership: "The challenges of the future will never be solved by the controversies of the past." Former Lib Dems were now beyond reach, and Labour's poll ratings tumbled again.

No wonder, then, the Greens enjoyed a surge in popularity, boosted by ex-Lib Dem voters and disillusioned Labour supporters. But Ukip too ate into Labour's support by attacking an out-of-touch, arrogant political elite that Miliband seemed to personify. With Labour bleeding support to the Greens and Farage's party, it was a mortifying blow to Labour when Ukip won the Wythenshawe byelection in early 2014.

Commentators began to look back to Ed Miliband's leadership campaign. He may have the similar wonky, geeky traits to his brother, they said, but perhaps he would have at least kept the party together and taken the fight to the Tories. As a divided, nearly-bankrupt party lurched towards electoral defeat, it became almost a cliche to speculate that maybe, just maybe, Labour had picked the wrong brother.