The prospect before the Labour Party has changed very dramatically since the start of the year. Instead of a hard but manageable slog to overtake the Conservatives as Britain’s largest party at the next general election, Labour politicians now contemplate a dismal scene of long-term exclusion from government: from the government, that is, of the rump UK, minus Scotland. The reason is that David Cameron, attempting to outflank the SNP during a midwinter silly season offensive, has managed to provoke Scotland’s first minister, Alex Salmond, into a daring dash for independence.

Salmond is not – as the English fail miserably to understand – an out-and-out nationalist. He is a gradualist in a party that has long contained divisions between fundamentalist and gradualist wings. He is cautious and canny, and, though an instinctive gambler and opportunist, does not take ridiculous risks. Salmond’s party is notionally committed to independence, or more properly to independence within the European Union, with the nations of Scotland and England remaining part of what he calls a ‘social union’: an updated version of the loose 17th-century Union of the Crowns, which also seems to hint at common standards of welfare provision. Scottish voters don’t care for outright independence and, understandably enough in an era which has seen major financial, security and environmental threats, prefer to be part of a large and resilient unit, though one which respects Scottish sensitivities. Salmond is crafty enough to shift easily between the high-flown rhetoric of nationalism and the reassuring language of unionism. He championed the EU against the folly of Cameron’s veto last December, and more recently presented himself to an English audience as the only party leader who could be trusted to defend British social democratic ideals. Might the English too be better off in Salmond’s social union?

Cameron’s bullying insistence that he, and not Scotland’s devolved government, will decide when, and in what manner, the Scots can hold a referendum on their own destiny, was constitutionally proper but came across as a cackhanded, sub-Machiavellian ploy. Salmond rejected Cameron’s gracious offer of a referendum on independence in 2013 and instead moved to hold one in 2014, the 700th anniversary of Robert the Bruce’s victory over the English at the Battle of Bannockburn. Yet until Cameron’s intervention it wasn’t clear that Salmond was leading Scotland towards independence at all. While independence might have been a very long-term goal, he seemed happy enough navigating the process of gradual disengagement and growing autonomy, and not overly concerned about reaching his notional destination. After all, the SNP’s leaders are sufficiently well versed in the jurisprudence of their late colleague Neil MacCormick, the professor of public law at Edinburgh and another gradualist, to perceive the ‘post-sovereignty’ parameters of 21st-century interdependence.

But did Cameron really blunder? Probably, insofar as he takes seriously his responsibilities as leader of the Conservative and Unionist Party; but he was almost certainly aware that he enjoyed comprehensive insurance coverage in the case of a mistake. If he did indeed jolt the Scots into leaving the union, there might be some red-faced disgrace for an Old Etonian traditionalist in the short term. But he also knows that his party is evolving into a Little Englander grouping. Not only are the Conservatives abandoning the ideals of a party which came close to rebranding itself as the Union Party in the middle of the 20th century, but Tory psephologists foresee a golden future as the natural party of government in the rump UK. Without support from its Scottish heartlands, how often, if ever, could Labour hope to form a majority in England, or even in England and Wales (with Northern Ireland returning its own non-GB parties to Westminster)?

The Scottish Question, which appears to have emerged suddenly in the last year, is an unexpected by-product of Labour’s own devolution arrangements. New Labour’s Scotland Act of 1998, founded on the ‘settled will’ of the Scottish people as expressed in the referendum of 1997, was supposed to strengthen the union by killing nationalism in Scotland ‘stone dead’, in the words of the Labour politician George Robertson. Under the Additional Member System introduced for elections to the Scottish Parliament, regional list MSPs are elected on a formula designed to correct the unrepresentative results which arise from the use of first past the post elections for constituency MSPs. The procedure was intended both to rectify the huge imbalance towards Labour in the constituencies, and to prevent any one party, whether Labour or the SNP, acquiring an elective dictatorship within the unicameral devolved parliament. Labour-Liberal coalitions prevailed – as expected – from the first elections in 1999 until 2007. The 2007 election threw up a minority SNP government, though this was a precarious affair, for the SNP had only one more MSP than Labour. In the run-up to the 2011 elections it seemed that Labour would reclaim its position as Scotland’s natural party of government. Two months before the election it led the SNP by 44 per cent to 29 per cent in the opinion polls. Instead, in the final weeks the Labour vote mysteriously imploded, and the SNP achieved what the AMS voting system was designed to thwart: a single-party majority in the parliament.

But Labour’s Scottish Question did not come out of nowhere. For decades there have been unpropitious auguries, which troubled prophets and doomsayers but caused only momentary alarm within the party. Indeed, the history of Labour in Scotland over the past half-century is a curious one of punctuated equilibrium, of long periods of complacency and uncomplicated electoral success interspersed with short squalls of panic. Until the Hamilton by-election in 1967 the SNP had won only a single Westminster seat, in a by-election protest vote in 1945. But Winnie Ewing’s victory in Hamilton took a very safe seat out of Labour’s grip. The defeat, according to a contemporary Nationalist observer, Oliver Brown, sent a shiver along the Labour benches ‘looking for a spine to run up’.

The Scottish Labour vote was managed at this point by Willie Ross, who was secretary of state for Scotland between 1964 and 1970, and again from 1974 until the retirement of his patron, Harold Wilson, in the spring of 1976. Ross came from Ayr in the Lowlands and had been a schoolteacher before rising to the rank of major in the Highland Light Infantry during the Second World War. He was also a pillar of the Kirk. A glowering – and sometimes snide – authoritarianism was second nature to Ross, who ran Scotland in the manner of a brusque, no-nonsense dominie. His ignorant charges included most Scottish Labour MPs, especially the new generation with radical tendencies, longish hair or flowery shirts, and SNP supporters, who were the worst kind of juvenile delinquent. Yet however dismissive he was in public of the ‘Nats’, Ross had an ultra-nationalist conception of how Scotland might quietly flourish within the union. There was no need at all for a Scottish parliament. What would a white elephant like that do for Scotland that couldn’t be done in private? Much better, surely, to have a fierce Scot in the cabinet who, in return for doing the prime minister’s bidding, was indulged in his demands, however outrageous, for cross-border subsidy.

Wilson was more worried than his henchman by the advance of the SNP. He set in train a Royal Commission on the Constitution, chaired first by Lord Crowther and then, after his death, by the Scottish judge Lord Kilbrandon, which came out in favour of a devolved Scottish assembly. In 1973, the year Kilbrandon reported, another major tremor ran through Scottish Labour politics: the victory of the SNP’s Margo MacDonald in a by-election at Glasgow Govan. Although Labour recovered Govan in the February 1974 general election, the SNP won seven seats. In the UK as a whole, Wilson had won 301 seats to the Tories’ 297. A minority Labour government would soon need to return to the polls, fearful that its Scottish vote might defect to the SNP.

During the first half of the 20th century Labour lent its support to the ideal of Home Rule for Scotland, but that commitment dwindled in importance behind the demands of central economic planning and UK-wide nationalisation, and in 1958 the party formally abandoned Home Rule. In 1974, Wilson decided that Labour needed to reverse course in order to neutralise the Scottish Question. Unfortunately, Labour in Scotland, in thrall for so long to Ross’s brand of Nat-bashing and the UK-wide cause of working-class brotherhood, was lukewarm about resurrecting the cause of Home Rule.

The matter was referred to the party’s Scottish Council Executive, which met on Saturday, 22 June 1974, the day Scotland’s football team played Yugoslavia in a televised World Cup game. Although the meeting was quorate, only 11 members of the Council Executive – there were more than thirty – turned up, and voted by six to five to reject a devolution platform. Wilson was appalled. Labour in London decided that Home Rule would have to be imposed on the Scots whether they liked it or not. On 24 July, the National Executive called on the party in Scotland to reconsider its position, and on 17 August, a special meeting of the Scottish Council of the Labour Party was held at the Dalintober Street Co-operative Halls in Glasgow. This is still remembered in the folklore of Scottish politics as the Dalintober Street fix. The trade unions, gently guided by Alec Kitson, deputy general secretary of the Transport and General Workers’ Union, persuaded Labour in Scotland to see the wisdom of London’s wheeze: the Scots – after a bit of prodding – would demand their own devolved assembly.

Labour has never quite exorcised the shade of Dalintober Street: the notion that Scottish Labour is subservient to London’s wishes – even, or perhaps especially, when the Scots are supposed to be doing their own thing. During the mid-1990s, George Robertson, the shadow secretary of state for Scotland and a veteran of the 1974 debacle, performed a series of craven contortions as he compliantly followed Tony Blair’s mazy hesitations on devolution. Blair was an instinctive centraliser and control freak, who breezily likened the projected Scottish Parliament to an English parish council, yet in his early years as party leader also felt constrained to observe the Home Rule commitments of his predecessor, John Smith. Of course, once devolution had been achieved, matters were simpler: as members of a UK-wide party, Labour politicians in the new Scottish Parliament were expected to follow the party line dictated by Blair from London, notwithstanding the supposed autonomy enshrined in the devolved settlement.

Unsurprisingly in the circumstances, the leading members of Labour’s ‘Scottish Raj’ decided to continue their careers at Westminster rather than become jumped-up parish councillors in what Billy Connolly called Scotland’s ‘wee pretendy parliament’. Of the party heavyweights, only Donald Dewar, a necessary replacement for Robertson, who had performed one obsequious gyration too many as shadow Scottish secretary, returned to a political career in Edinburgh. There was no reservoir of talent on which to draw when Dewar, the father of the Scottish Parliament, died suddenly in 2000 after slipping on steps in Edinburgh and hitting his head as he fell. No cabinet member came north to replace him. A succession of lightweight Scottish Labour leaders followed in quick succession. Henry McLeish had to step down after a scandal over the subletting of his constituency office; Jack McConnell resigned after Salmond defeated him in the 2007 election. His successor, Wendy Alexander, the sister of the current shadow foreign secretary, Douglas Alexander, and a protégée of Dewar’s, was the intelligent moderniser the party needed. But she got caught up in an exaggerated fuss about the financing of her leadership campaign, which had received an ‘impermissible donation’ of a mere £950 from a businessman based in Jersey. This molehill forced Alexander out of the leadership. Thereafter, Iain Gray, a sober but uncharismatic politician with an unfortunate surname, led Labour to humiliating defeat in the 2011 election. The campaign was badly managed: Labour focused its attacks on the Tories at Westminster rather than on the party’s real rivals, the SNP government in Edinburgh. Moreover, a number of rising figures had complacently failed to insure themselves against defeat in their constituencies by taking places high on the regional lists as the AMS voting system allows, and this led to a further cull of potential leadership candidates. Now Gray has been replaced by Johann Lamont, who will find that she too has trouble building a profile in the Scottish media when the large and familiar personality of Salmond casts such a dark shadow over all his rivals, in other parties as well as his own.

In part, this is for Scottish Labour a sad saga of accident and contingency, stretching all the way back to the premature death in 1994 of John Smith, a committed devolutionist since the mid-1970s and a more sympathetic patron of Home Rule than Blair. However, there are also more fundamental problems with Labour in Scotland. The party has had a dual function, as a platform for politicians set on high-flying Westminster careers, and as the domestic preserve of what is known as a ‘numpty’ tendency of unthinking and subservient local government apparatchiks, some of whom have been lucky enough to win promotion to the devolved parliament. That some of the sharpest thinkers among Scottish Labour politicians have been awkward anti-devolutionists, such as Brian Wilson, the founder of the West Highland Free Press and campaigner for land reform, and the maverick Tam Dalyell, only serves to sharpen the contrast with the numpties. Moreover, there has long been a suspicion – more than that perhaps – that Labour in Scotland takes for granted the downtrodden working class it claims to represent. In recent decades, Tommy Sheridan, the populist leftwinger who has just got out of prison after serving time for perjury, has tried – not unsuccessfully – to wrest the votes of the dispossessed and neglected on Glasgow’s council estates away from a complacent Labour machine to an authentically socialist alternative. The problem for Labour is not so much that Sheridan or Salmond is about to capture the bulk of the Labour vote, as that the Labour machine has anaesthetised its own constituency and New Labour has alienated it, bringing otherwise safe working-class constituencies, with low turnouts, into play.

Clearly Labour is not in a strong position to lead the unionist case in the Scottish referendum, but no other anti-nationalist party enjoys the same breadth of support or has leaders with positive name recognition in Scotland. It seems likely that Alistair Darling, the former Labour chancellor, will play a prominent part in the anti-independence campaign. But much more worrying for the party is its future in England. The referendum isn’t likely to solve Labour’s English Question: how can the party connect in a positive way with Englishness? Devolution has let daylight in on the dark arts of cross-border subsidy – Ross’s nightmare – and ignited the passions of a hitherto passionless English nationalism. Devolution allowed the Scots to make their own democratic decisions about how to spend their block grant from Westminster. Now the scale and indeed the existence of the block grant itself are open to question.

Scotophobia, which last featured as a significant political force in the agitations of the 1760s against the Scottish prime minister Lord Bute, has resurfaced as a respectable element in the populist conservatism of Middle England. The likeliest beneficiaries of the new English nationalism are Ukip and the Tories. Indeed, dislike of the devolution settlement seems often to complement the Euroscepticism of the Tory right. Just as Little Englanders forget the benefits they derive from the EU but feel entitled to whine about the payments the UK makes to Brussels, so they are oblivious of the mountain of tax receipts received over several decades from North Sea oil or the lavish, but partially concealed, subsidies that find their way to London and the capital’s hinterland. Labour has a further problem. If, as seems possible, the UK might be heading for a Czecho-Slovak style divorce, with the core of the union showing the door to its poorer periphery, then Labour might have a long-term English problem were it to appear too obviously the pro-Scottish party of union.

However, there are also many ethnic Britons in English cities who do not think of themselves as ‘English’ and identify the label with racism and xenophobia. If the Anglo-Scottish union does break up, such groups will feel a further degree of alienation from a country they will have difficulty naming as home. There is a respectable case to be made for British Unionism as something quite distinct from say, the atavistic British Protestantism of the Ulster Unionists. Nevertheless, Labour is not yet able to make an informed choice on the English Question. Labour and the Conservatives do not diverge simply as a party of the left and a party of the right. While the Conservative and Unionist Party has too often been obsessed about the constitution and nationality, not least as these concerned Northern Ireland or Europe, Labour’s primary focus on class and the redistribution of wealth across the UK as a whole has led it to pay scant attention to the multinational union-state itself, except insofar as that state functions as an instrument of social and economic policy. But the moment has come when Labour might have to ask if its commitment to Britain – understood as sotto voce code for reliance on its Scottish heartlands – is undermining its appeal in Middle England; or whether a rather different articulation of multiethnic Britishness, either in the UK or in a rump England, might come to trump a distorted version of what it is to be English.