Looking back, the tip-off that Labour under the leadership of Ed Miliband was heading for a fall wasn’t any failing in his personal performance during the campaign. His flirtation with Russell Brand may have been ineffectual and demeaning, and the plinth recording his meaningless pledges was an idea straight from one of the meetings of uncomprehending minds portrayed in the magnificent BBC parody W1A. Taken as a whole, though, the campaign that Miliband waged turned out to be surprisingly good, and not only because it was fought against a background of low expectations. He proved a better communicator than many believed possible, and eerily calm under fire. The problem lay with the message that the former Labour leader had decided to deliver.

As was clear before the election campaign began (see my essay “Misunderstanding the present”, from the New Statesman in February 2015), Miliband’s message was directed not to any country that exists, but to some quite different land that he and his chosen advisers had persuaded themselves was coming into being. Convinced that the attitudes and values that enabled Margaret Thatcher, and later Tony Blair, to win three successive elections had been discredited by the financial crisis, Miliband staked Labour’s future on the wager that the triumph of market individualism in Britain could be reversed. The bet paid off only in Scotland, and there it was not Labour (which suffered a wipeout that is probably irreparable), but the SNP that pocketed the winnings.

Miliband’s gamble was based on an image of contemporary Britain that screened out some of its most defining characteristics. Both before and after the referendum on Scottish independence, he and his advisers saw the fact that Britain is a multinational polity as essentially insignificant. They did not perceive the increasing fragility of the British state. They continued to regard the central divisions of British society as based in class, when class identities were rapidly mutating and in some cases losing their primary importance. They continually invoked the dangers of growing inequality, but their focus was on issues such as non-doms and bankers’ bonuses rather than the far more damaging divisions that have resulted from whole sections of society being left behind by the effects of unchecked globalisation. The enormous difficulties of governing a country where competing nationalisms jostle with market individualism and the socially disruptive effects of global market forces were not even dimly understood.

***

For me the first clear sign that Ed Miliband was gripped by a delusional view of the country he wanted to govern came in a seemingly insignificant episode: the hullabaloo surrounding his hiring of David Axelrod over a year ago. The left’s obsession with US politics is hardly new. Tony Blair seemed always on the brink of addressing his audiences as “fellow Americans”, and even when Britain rejected New Labour a fixation with the American system continued unabated. Like others in his party, Miliband can’t help thinking that Britain would be far better off if only it could adopt what he thinks of as a more modern sort of politics: the type that is practised in the US, with its written constitution and its refusal to defer to antiquated institutions. This mindset is curious in a number of ways. It is true that, perhaps more than any other country in the developed world, Britain is an ancien régime – a construction of monarchy and empire. But it is also true that in some ways Britain is the most modern of countries, not least in its openness to globalisation, which is far greater than that of the US.

Labour’s fixation on an American model of politics might not matter if it didn’t have practical consequences, such as signing up Axelrod as a guide to fighting a British general election. The well-rewarded Obama adviser has been criticised for not being around much of the time, but perhaps this was just as well. Someone who just a few months ago could give a lengthy interview (Guardian, 15 February) in which Scotland wasn’t once mentioned may not be the most useful oracle. How could anyone whose skills were honed in such a different political culture be attuned to an unravelling of class and national identities and loyalties for which they have no instinctive feel? In the event, the prospect of a minority Labour government being kept in power by the SNP left Labour heavily compromised in the eyes of many English voters. Miliband responded by declaring that there would be no deals with the SNP. Unsurprisingly, no one believed him. Under the ruthless direction of the Tories’ Australian electoral strategist, Lynton Crosby, this proved to be one of the main catalysts for Labour’s defeat.

Axelrod’s blind spot in relation to Scotland reinforced that of Miliband and his British advisers. Labour’s collapse north of the border wasn’t the only reason for the party’s downfall. Even without that wipeout, David Cameron would still be in Downing Street. But the catalogue of errors and illusions that led to Labour’s Scottish debacle are instructive as to the enormity of the obstacles it will have to surmount if it is to recover as a party of government.

Labour is beginning an agonised search for some way back to the position it imagined it had in Britain before the 7 May election. But this is not 2010, when Ed Miliband was elected as leader. Nor is it 1983, when Labour condemned itself to almost a decade and a half in the wilderness. The threat Labour faces today is larger and more genuinely existential than it has faced at any time in its postwar history. When Michael Foot led the party to catastrophic defeat at the hands of Mrs Thatcher, Labour retained its working-class bastions in the north and Scotland. Today, though it has made no advance in terms of seats, Ukip has emerged as the third party in the UK in terms of votes – much larger than the Liberal Democrats, the SNP and the Greens – and in constituencies where it came second it might be capable of mounting a tough challenge to Labour in 2020. Labour’s climb-back will be made all the more difficult by changes to constituency boundaries (thwarted by the Liberal Democrats) that Cameron will bring in.

The working-class support on which Labour has relied is melting away in the north. In Scotland it is already gone. The SNP may not retain its current level of popularity for long. But when it proves unable to deliver on its promises, the opposition to it is more likely to be expressed in internal fissures and the formation of new movements (a process made easier by the proportional voting system for elections to Holyrood) than in any revival of Labour. Everything suggests that Scotland’s shift to having its own political culture will be irreversible.

No doubt the SNP’s triumph was made easier by errors on the part of the Westminster parties and more generally by the contemptuous indifference they displayed towards Scottish affairs. Thatcher’s use of Scotland as a laboratory for testing the poll tax (an experiment whose result she refused to accept) was a big factor in the destruction of Conservative unionism. Equally, Blair’s war in Iraq led to mass desertions from Scottish Labour, while the refusal of Labour’s London elite to face up to the decline in Scotland (against which this magazine warned even before the Nationalists’ victory in the 2011 Scottish Parliament election) demonstrated a refusal to take the SNP and, indeed, Scotland seriously.

The SNP’s skill in shape-shifting also played an important role. Nicola Sturgeon’s astonishing victory is a tribute to her outstanding leadership. It is worth noting the impressive ease with which the SNP has transformed itself from being a party with strong neoliberal tendencies – exemplified by Alex Salmond’s talk of an “arc of prosperity” in which Scotland would join Ireland and other European countries to pursue growth through low taxes and business-friendly policies – to an anti-austerity party of the social-democratic left. The SNP’s neoliberal past is now commonly forgotten (but not, it seems, by Rupert Murdoch).

Although the self-absorption of the English metropolitan elite has been a critical factor, it cannot fully explain the rise of the SNP. Scottish nationalism is not just a reactive movement, but one that mirrors developments in other European countries and other parts of the world. The progressive narrative of a few decades ago which anticipated that nationalism (along with religion) would have a dwindling role as a source of human identity never had much substance. A decade and a half into the 21st century, it’s a story that is proved false daily by events.

Partly in reaction against the upheavals of globalisation, the politics of identity is as potent – and as dangerous – as it has ever been. The SNP insists that its nationalism is not of the malign, us-and-them variety that has resurfaced in parts of the European continent. Yet the difference between civic and ethnic nationalism isn’t categorical or unequivocal; it has always been blurred and shifting. In effect, the SNP is buying in to another version of the progressive story – one in which nationalism doesn’t fade away, but can be purged of harmful effects without much difficulty and become the basis of a type of supranational government.

***

This progressive narrative may explain the remarkably sunny view the SNP takes of the European Union. Listening to Nicola Sturgeon insisting that Scotland’s future can only be as a fully paid-up member, you would never suspect that the EU was in the midst of the most intractable crisis in its history. Here, SNP thinking runs in close parallel with that of the Liberal Democrats and pretty much all of Labour: any problems the EU may be facing are temporary and surmountable within its existing structures. As Vince Cable put it, voicing the conventional wisdom in an interview with the BBC in 2011, “The eurozone project may turn out to be a great success.”

It should be obvious that this is not a claim based on observable facts or tendencies. It is a declaration of faith that the eurozone is leading the continent towards a higher form of political organisation. A contrary view is closer to the truth: a utopian version of the European project is taking the continent back to the toxic politics of the 20th century. Viewed empirically, the eurozone is a disaster, riddled with mass unemployment and containing some extremely smelly political movements. In what sense has France – where Marine Le Pen’s Front National may soon come within spitting distance of the Élysée Palace – achieved a higher form of modernity? The ramshackle United Kingdom may in the end turn out to be more durable, and more modern, than the EU’s crumbling, Byzantine structures.

These aren’t theoretical problems. They have a direct bearing on the unprecedentedly adverse environment in which Labour will struggle to reinvent itself. The party has positioned itself on the basis that the difficulties of the eurozone can somehow be resolved by moving towards fiscal union; but that involves movement towards political union, which is a chimera. If the crisis intensifies instead, with the stand-off over Greece persisting or the country being forced into a disorderly exit from the euro, the result can only be to strengthen Cameron. An early in/out referendum was never likely to result in Britain deciding to leave the EU. British voters will not take a leap into the unknown until they are sure the status quo is no longer viable. Brexit will be even less likely if Cameron is able to deliver an altered relationship with the EU.

There are many who assume that, with a majority smaller than the one John Major had after his unexpected victory in the general election of 1992, Cameron will be held hostage, as Major was, by the more extreme Eurosceptic elements in his party. No doubt his majority will be eroded over time, and if he stays the full five-year course he may well end by relying on the Democratic Unionist Party. But his situation at the start of his government is quite different from Major’s: rather than having to wage a trench war in order to secure greater integration with Europe through the Maastricht Treaty, Cameron is offering his party a greater degree of detachment from the EU.

It is true that the party is probably even more Eurosceptic than it has been in the past. However, the threat posed by Ukip is now much reduced, and may in future be directed against Labour, rather than the Conservatives. A referendum in which Cameron succeeds in delivering a vote to stay in the EU will leave him and the Conservatives stronger than they are now.

None of this is a foregone conclusion. Cameron may be unable to negotiate the looser relationship with the EU that most British voters would prefer. Extreme Eurosceptics in his party may prove irreconcil­able. The in/out referendum could go against him. Yet even in this scenario there is no comfort for Labour. A common view has it that Brexit will be followed by Scotland breaking away from the Union. Nicola Sturgeon seemed to endorse this view when she said that a British vote to leave the EU would trigger a referendum on independence in Scotland. But the movement towards Scottish separation may not be as unstoppable as it seems. The irony that momentum for independence is increasing against the background of a failed referendum less than a year ago has been widely commented on. But there may be a deeper irony looming on the horizon: while Scotland is for now a polity almost wholly dominated by a party committed to separation from the UK, another referendum on independence could still very easily be lost.

Neither Sturgeon nor Salmond fought the 7 May election on the issue of Scottish independence. What they seemed to want was something else, which they may have found more attractive: a turbocharged version of devo-max, with full control over tax and spending.

If Labour had emerged from the election as a minority government reliant for its survival on the SNP, some such settlement might well have been reached, though whether the SNP could have threatened credibly to bring Labour down in order to achieve it is doubtful. The paradox of the present time is that offering a bold form of devolution to Scotland serves the Conservative interest. Full fiscal control would give Scotland’s voters a taste of what independence would cost – particularly if it was accompanied by the scrapping of the Barnett formula, which evens out state spending on public services across the UK. It is doubtful whether a majority of Scottish voters will be willing to break the Union once they know what ultra-devo-max would mean for their pensions.

A future referendum on independence isn’t going to be determined by the result of an election fought under a first-past-the-post system. Even in the unlikely event of Brexit, it cannot be taken for granted that Scottish separation would follow. The crucial question of which currency a sovereign Scotland would use remains unresolved, while the volatility of the oil price casts an unsettling shadow. It shouldn’t be too surprising if the SNP leadership appears less than wholly committed to holding another referendum any time soon.

Cameron may be at risk of being impaled on the interlocking spikes of Scotland and Europe. But it is equally possible, and perhaps more probable, that he will achieve what he may always have wanted: devo-max for Scotland in the UK and for the UK in Europe. Where will Labour be then?

***

Displaying a distinct lack of ceremony, several of Miliband’s colleagues have already shown themselves ready to step into his shoes. Something like a consensus seems to have emerged, suggesting that Miliband’s mistake was in ditching the New Labour inheritance and tilting the party from the centre to the left.

While ruling himself out from the current struggle for the succession, David Miliband expressed the same view. Peter Mandelson has observed that the hole at the heart of Ed Miliband’s programme was the lack of an economic policy. These are more serious responses to the cataclysm that has befallen the party than Neil Kinnock’s reaction on election night, when he seemed to blame the electorate for allowing itself to become the puppet of demonic tabloid newspapers – a view reminiscent of Bertolt Brecht’s suggestion that the simplest solution to the workers’ uprising in East Germany in 1953 would be for the government to dissolve the people and elect another. Labour’s downfall resulted not from popular false consciousness, but from the hubristic self-delusion of its leadership.

But the belief that Labour’s future can be secured by reverting to Blairism is hardly less wishful. Waffling on about aspiration and wealth creation won’t reproduce the electoral success the party enjoyed under Blair’s leadership. History can’t be rewound in this way. The foul stench of the Iraq war will hang over any attempt to rehabilitate the Blair years, and the sense of prosperity in those times was generated mostly by debt and rising house prices.

Politics has in any case moved on since then. David Cameron’s Conservative Party is itself the product of a Blairite modernisation. How would a Blairite restoration that rebranded Labour as Cameron-lite give the party what it now needs: a compelling reason why British voters should entrust their future to it rather than to the party that is in power?

Miliband staked his party on an unreal vision of Britain. His uncanny serenity ­during the campaign showed a quality he has in common with Blair – a capacity for certainty, enabling him to think that what he wants to believe must be the case. Redefining Labour will take more than one new leader, but none of them would be able to bring back a country that did not exist. If Miliband’s successors are ever to be in a position to change Britain, they will first have to be willing to understand it.

John Gray is a contributing writer for the New Statesman. His latest book is “The Soul of the Marionette: A Short Enquiry into Human Freedom”