It was meant to be different this time. From its defeat in 2010 until the moment the polls closed on election day, Labour believed that it could “short-circuit” history by returning to government after a single term in opposition. But, as in 1955 and 1983, a bad election result has been followed by a worse one. It is David Cameron, not Ed Miliband, who has defied historic precedent. The Prime Minister is the first incumbent since Lord Salisbury in 1900 to increase his party’s vote share after serving a full term in office. Throughout its campaign Labour repeated the assertion that the Conservatives could not win a majority. They did.

Labour had relinquished hope of becoming the single largest party before election day – its private polls consistently showed it performing worse than those publicly available (just as the Tories’ showed them exceeding expectations). But it clung to the hope that it could enter power by virtue of Miliband being the only leader capable of commanding the confidence of the ­Commons. Three days before the election, Labour aides briefed me and other journalists on the finer details of the Cabinet Manual. One source spoke of how some in Labour had became “experts” in Ramsay MacDonald’s 1924 administration: the last time a second-placed party took office.

When the BBC’s exit poll was published at 10pm on 7 May, Miliband was at his constituency home in Doncaster with Bob Roberts, his director of communications, and Stewart Wood, his intellectual consigliere. He reacted with incredulity to its projection of 316 seats for the Tories and 239 for Labour, crying aloud that it must be wrong. Back at the party’s London HQ in Brewer’s Green, Charlie Falconer, who was overseeing preparations for government, sought to assuage distraught staffers with a rousing speech, assuring them that exit polls had been mistaken before. Labour’s spin operation was instructed to rubbish the numbers to journalists. “We are sceptical of the BBC poll. It looks wrong to us,” a text message read.

But none of the early results contradicted the forecast and the mood turned to despair when Nuneaton, a key Labour-Tory marginal, declared at 1.51am. Far from showing a swing towards the opposition, it showed a swing towards the Conservatives. At this point, concluding that the game was up, ­Labour staffers took solace in drink. “Every­one got hammered,” a source said. The talk at HQ turned to Miliband’s now inevitable resignation. By 2.34am, after a silence of more than two hours, the party’s spin team all but conceded defeat, warning that “the next government will have a huge task uniting the country”. Miliband’s speechwriter and university friend Marc Stears drafted a resignation address, which no one had prepared before that point. After travelling from Yorkshire to London, Miliband announced his departure to tearful staff at 9.45am on Friday, drawing on Ted Kennedy’s oration at the 1980 Democratic National Convention (“The dream shall never die”). At 12.12pm, he delivered a near-identical speech to journalists at One Great George Street in Westminster, and closed the curtain on his five-year leadership of the party.

Labour now finds itself in the foreign land of a Conservative majority – an outcome that few ever contemplated. But there were some prescient MPs. During the last and greatest crisis of Miliband’s leadership, last November, several predicted a Tory majority to me. When I reminded one of this, he replied: “I did put some money on it, so I can give my party a slap-up dinner.”

Many believed that Labour would always struggle to win if it trailed the Conservatives by a double-digit margin on both leadership and economic management (a position from which no opposition has ever won). From this perspective, the election was lost long ago. The Tories’ warnings of a Labour-SNP alliance helped to lure previously resistant Ukip and Liberal Democrat supporters into their camp. But it only proved so lethal because it preyed on existing doubts about the party. The framing of Miliband as the puppet of Alex Salmond and Nicola Sturgeon reinforced the impression of him as weak. The cry that a Labour-SNP partnership would lead to higher taxes and higher borrowing confirmed the view of the opposition as fiscally reckless. To win again now, the party needs to master what Tory strategists referred to as “the fundamentals”: strong leadership and economic credibility. This is less a matter of being more left-wing or more right-wing than one of simply being better.

But the heterogeneous character of the party’s defeat precludes easy definition. It lost votes to different groups in different regions for different reasons. Anti-­austerity Scots, anti-immigration northerners and fiscally conservative southerners all turned against Labour. It is hard to appease one group without simultaneously alienating another. MPs are able to cite whichever results suit their ideological predilection. The anti-austerity and anti-Trident left points to the calamity in Scotland. The anti-immigration and Eurosceptic right warns of a similar fate in the north (where Ukip finished second in 19 seats). The Blair-type reformists cite the south (where the party lost seats to the Tories) and appeal for fiscal restraint and an embrace of enterprise.

There is no cost-free approach. The task for Labour is to resolve which is the least costly. It is the modernising leadership contenders – Chuka Umunna, Liz Kendall and Tristram Hunt – who have moved fastest to define the defeat (one rival campaign told me they were behaving like “family members taking jewellery off a corpse”). They recognise that the decisive nature of the loss aids their cause. No one can argue that with “one more heave” Labour could have got over the line. Even without the loss of 39 of its 40 Scottish seats, it would have finished 60 behind the Conservatives.

Few believe that the party can transform its performance north of the border, where the shift towards the SNP is structural rather than merely cyclical, in a single parliament. Many agree, as a shadow cabinet minister puts it, that “the route to power lies through Middle England”. It is here, as in 1992, that the election was lost. To win, Labour will need to make large gains from the Tories. Umunna’s decision to launch his campaign in Swindon, a Conservative-held seat, was symbolic of his focus on this task. His supporters regard Miliband’s limited effort to win over Tories as one of his biggest strategic failures. As an aide is said to have remarked on election night: “Who are these people who vote Tory? I’ve never met any of them.”

Unlike in 1994, when there was only one possible victor in the leadership contest (Tony Blair), and unlike in 2010, when there were two (the Miliband brothers), there are several serious contenders. Andy Burnham, who has polled consistently as members’ favourite shadow cabinet minister, and who delivered the best-received speech at last year’s conference, should not be underestimated. Under Labour’s preferential voting system, the winner – as in 2010 – will be the candidate best able to appeal across factions.

Whoever triumphs faces a task even more daunting than that of Miliband in 2010. Labour needs 94 gains to achieve a majority, a feat that only the Liberals in 1906 and Labour in 1945 have achieved from a starting position so weak. To add to this arithmetical Everest, the Tories will use their new-found majority to pass the constituency boundary changes previously vetoed by the Lib Dems, increasing their standing by at least 20 seats. At the next election, whether in 2020 or earlier, Labour will also have to contend with a new Conservative leader who may revive the party’s support just at the moment it is flagging (as John Major did in 1990).

But MPs are consoling themselves with the thought that if a week is a long time in politics, five years is an eternity. Just months after their victory in 1992, the Tories’ economic reputation was eviscerated by Black Wednesday. The scale of spending cuts, the risk of a housing or banking crash and possible EU withdrawal all make it impossible to rule out a similarly epochal event. If, as in 1994, Labour elects a leader with wide-ranging appeal, it may be able to achieve a majority. The lesson of this election, which almost all called wrong, is never to dismiss what is thought impossible.