The story of the Labour Party runs deep through the modern history of Scotland from the late 19th century. Several of Labour’s most influential founding fathers came from north of the border, including Keir Hardie, the first leader. From 1945 until recent years, the party in Westminster was always supported by a phalanx of loyal Scottish MPs. From the 1960s, Labour possessed overwhelming political authority and influence in Scotland, at both local and national levels. At the end of the century, it was the only force capable of delivering a Scottish parliament, which it did successfully in 1999. As recently as the 2010 general election, that historic hegemony in UK elections was maintained, despite the party’s narrow defeat at Holyrood in 2007.

But then, since the elections to the Scottish Parliament in 2011, when a mortal enemy, the Scottish National Party, achieved the unthinkable by winning an overall majority of seats, Labour has been traumatised by one catastrophe after another. Large numbers in several of its long-established heartlands voted for independence in 2014 against Labour policy and while its membership stagnated or fell in some areas that of the SNP soared to more than 115,000 by early 2016. In the general election last year, Labour was comprehensively routed, and just escaped annihilation by winning a single parliamentary seat out of the 59 in Scotland (the SNP won 56).

Since then, despite new leadership and some fresh policies, the party continues to languish in the opinion polls as the May Scottish Parliament elections loom closer. Even the left-leaning Jeremy Corbyn’s victory in the Labour leadership race seems to have had little effect to date in persuading the disenchanted to return to their old political home. A poll last month suggested that only 22 per cent intended to vote for Labour in the constituency section of the Holyrood election in May, compared to 53 per cent for the SNP. Yet, at the same time, two out of five Scots said that they approved of the party’s daring plans to raise income tax by a penny in order to protect health and social services, in an attempt to outflank the Nationalists on the left.

Some merchants of doom predict that the Conservatives could push Labour into third place in Scotland, and a few alarmists even contend that the party may face extinction north of the border.

These disasters did not come as a series of thunderbolts from a clear sky. Instead, it would be more accurate to consider the fall of Labour in metaphorical terms, with the party in the form of a large nut, still shiny and well preserved on the outside but steadily rotting over many years on the inside until, when external pressure was applied, disintegration followed.

The end of the golden age of Scottish Labour came about in part because of the gradual erosion and, in some cases, the disappearance, dating back to the late 20th century, of the historical pillars on which its power had been built. These foundations were both structural and ideological. Labour had originally been born out of the trade union movement; the collapse of union membership, with the loss of the old mining and manufacturing industries in the 1980s, was bound to have a negative impact. Over half of the Scottish workforce was unionised in 1980, yet by 2010 that figure had fallen to a third. The new service-, oil- and finance-based economy, outside the public sector, was markedly less unionised than before.

Moreover, council housebuilding in the 1950s and 1960s had created huge Labour fiefdoms in and around the Scottish cities. The Tories’ “right to buy” was just one (but nonetheless an important) factor that caused their later contraction. By 2005, council housing had fallen to 15 per cent of the total stock. As late as 1981, most of the housing in 40 of Scotland’s 71 constituencies was publicly owned. By 2010, not one of them had a majority of council tenants.

Closely allied to council housing was the Labour empire in local government. The first-past-the-post electoral system had long resulted in Labour domination of many local authorities: in 1995 the party governed 20 of the 32 councils in Scotland. Hegemony often grew into self-perpetuating oligarchy, leading to accusations of corruption, croneyism, nepotism and the malign influence of patronage networks. In parts of west-central Scotland, opposition councillors became almost an endangered species as councils began to resemble one-party states in which the important ­decisions were made by Labour groups behind closed doors.

Ironically, it was the Jack McConnell-led Labour administration in the Scottish Parliament between 2001 and 2007 that ended these monopolies by abandoning first-past-the-post in local elections and introducing proportional representation. A more democratic spirit soon began to flourish in town halls across the land and the SNP was the party that most enthusiastically exploited the new opportunities.

Then there was the problem of ideology. The transformation in the 1990s of old Labour into New Labour, which seemed to embrace a free-market philosophy and failed to reverse Thatcherite reforms, triggered much disenchantment among the party’s supporters in Scotland. Surveys taken after the 1999 Holyrood election concluded that less than half the respondents thought that New Labour looked after Scottish interests. Similar evidence covering the period between 1997 and 2001 demonstrated declining support for the proposition that New Labour looked after class and trade union concerns but increasing agreement with the view that the party primarily looked after the concerns of business and the affluent in society.

The free-market rhetoric emanating from London on public-sector reforms in the National Health Service and education did not go down well, especially when Labour in Scotland also started to speak about possible private-sector involvement in the NHS north of the border. The decision to take part in the Iraq War in 2003 engendered much hostility and outright condemnation from Labour ranks in Scotland and alienation intensified as a result.

McConnell, the then leader of Scottish Labour, conceded: “Alex Salmond’s consistent opposition to the Iraq War started to become the accepted public opinion and left what we were doing on the inside pages of the papers, rather than the front.” He might

have helped his cause by distancing the Scottish Executive from New Labour policy on Iraq, in the same way as Welsh Labour’s then first minister, Rhodri Morgan. But McConnell was apparently too concerned that he might antagonise the Labour leadership in London and so harm party unity.

One of the old bastions of Labour dominance that had provided cast-iron support for the party since the 1920s also began to weaken from the 1990s. Since the foundation of the Irish Free State in 1921, the Irish-Catholic diaspora had transferred its allegiance en masse to Labour from the Liberals. It was thought that the “people’s party” would provide initiatives of social justice and amelioration to one of the most disadvantaged communities in Scotland.

That bond, once established, endured over generations. The descendants of Catholic immigrants produced councillors, MPs, party workers and trade union officials in large numbers who supported the Labour cause. Even when devolution became a serious political issue from the 1960s onwards, the community’s loyalty remained rock solid. Whether rightly or wrongly, the SNP was regarded as a “Protestant party”, at a time when sectarianism in many working-class communities was still rife.

Yet even here, among the most loyal of the loyal, erosion was starting to occur. By 2000, Scots from an Irish-Catholic background had climbed the ladder of the new educational opportunities and achieved occupational parity with their fellow citizens. Several had managed to occupy some of the highest positions in the land in the judiciary, government and the universities. The community was becoming more integrated into Scottish society. At the same time, the SNP openly courted the Catholic vote with lavish

praise for the system of high-performing denominational schools. This was the background to the substantial Catholic vote for independence in September 2014.

***

Much loyalty to Labour from working-class communities in the past had depended on its provision of public services and low-cost housing. Areas existed in west-central Scotland where, at the start of the new millennium, the public spend came close to 75 per cent of the local economy. This pattern was not simply confined to parts of Glasgow, but stretched into the hinterland of the city across the counties of Ayrshire, Dunbartonshire and Lanarkshire. This region had been the dynamic heart of the old industrial economy, its modern ills a result of the social consequences of deindustrialisation.

Its demographic weight needs also to be emphasised. Greater Glasgow has a population of 1.2 million – it is the largest urban area in Scotland and the fifth-biggest in the UK. The populations of the three surrounding counties number 1.1 million combined. In all, the region represents 44 per cent of the population of Scotland. It would be absurd to suggest that every village, town and district within it were a centre of deprivation. Indeed, East Renfrewshire and several towns and districts such as Bearsden, Newlands, Alloway in Ayr, Troon and many other places boast standards of living equal to those in the wealthiest parts of the UK. Yet right next to them are areas of deep, historic relative poverty and social disadvantage.

Because of the high expenditures on them from the public purse, these poor localities have bred a different attitude to the state compared, for instance, to that of most of the population of the Home Counties and the south-east of England. In many of the working-class communities in Glasgow, areas of west-central Scotland, Dundee and parts of Edinburgh and Stirling, there is no equivocation about the vital importance of public funding and need for state support.

Meanwhile, the New Labour project was launched to respond to the changes in political attitudes in “Middle England” that had occurred over many years of Conservative rule. However, as Labour steadily moved away from left-leaning policies, it faced the danger and then the reality of losing touch with some Scottish working-class communities that were not in any way alienated from collectivism and state control of public services. As the party began to concede this ground, the SNP moved in to fill the political gap. It was an opportunity that the Nationalists grasped with relish.

After they assumed power in Holyrood as a minority government in 2007 (the change of name from “executive” was significant), they soon showed themselves to be a formidable foe. Their opponents were no match for the talent of the SNP front bench, ably led by Alex Salmond as first minister. All of Labour’s Scottish “big beasts” after 1999 – apart from Donald Dewar – had decided to continue to pursue their careers on the bigger stage of Westminster.

That pattern seemed to confirm the belief that Labour in London regarded Holyrood with a mixture of contempt and condescension and that the parliament in Edinburgh was considered as having little more status than a parish council. One observer cruelly commented that, because of Labour’s focus on Westminster, its Holyrood representation, “with very few exceptions, [involved] a cohort of shifty election agents, superannuated full-time trade union officials and clapped-out local councillors”. It was one element in an emerging consensus that for too long Labour had taken its voters in Scotland for granted.

The SNP government soon gave the Scots what many of them craved: a return to old Labour policies, with the removal of prescription charges, the scrapping of tuition fees for university students, a freeze on council tax and the abolition of tolls on the Forth and Tay road bridges.

These decisive interventions contrasted with the mediocrity of the Scottish Parliament’s previous executives, led by Labour in coalition with the Liberal Democrats. At last, it seemed, the real potential of devolution was being demonstrated for all to see. A clear contrast began to emerge between the new government and what had gone before. The pre-2007 executive was led by Jack McConnell (now Lord McConnell of Glenscorrodale) as first minister. It was not entirely devoid of initiatives, which ranged from a ban on smoking in public places to attempts to grapple with Scotland’s age-old sectarian problem. But there was still a sense of unfulfilled expectations and a failure to exploit the potential of the parliament after the long, hard struggle that had led to its establishment.

The SNP was not without critics, who argued with some justification that Scotland’s problems of social inequality and gross disadvantage were neglected. But to many at the time, the new government was a breath of fresh air, not because there was any yearning yet for Scottish self-determination, but because it delivered sound administration and some popular policies.

The SNP possessed two crucial political advantages over Labour. First, given its raison d’être, the party was seen as the champion par excellence of Scottish interests. Second, it had the freedom of being its own master in its own house. Scottish Labour, on the other hand, was an integral part of the broader British Labour movement and there was more than a suspicion that London called the shots north of the border. That view seemed to be confirmed when the party’s former leader in Scotland, Johann Lamont, gave her resignation speech in October 2014. She declared that the UK Labour leadership treated the Scottish party as “a branch office”, and proceeded to make the damaging allegation that Scottish Lab­our had been prevented by its masters in the south from introducing the kind of social-democratic policies that might have stemmed the tide of SNP electoral success. The Nationalists had won a historic landslide victory in Holyrood in 2011, which opened the way for the referendum on Scottish independence in September 2014.

***

There is a remarkable irony to this turn of events. In the 1990s, the then shadow secretary of state for Scotland, George Robertson (now Lord Robertson of Port ­Ellen), asserted: “Devolution will kill nationalism stone dead.” Several years after the foundation of the Scottish Parliament, another well-known Labour figure – Brian Wilson, the former MP and minister who is an arch-opponent of devolution – surveying the wreckage of his party in the Scottish elections, concluded that in delivering the parliament, Labour had, in effect, constructed its own scaffold.

Wilson had a point. Holyrood did help to swing the focus from class to identity politics, because almost all issues debated there had a Scottish provenance. More crucially, the new system of proportional representation held out electoral opportunities for the SNP. Since its entry on to the modern political scene following its victory in the Hamilton by-election of 1967, the party had struggled to gain more than a handful of seats in UK general elections, even when it won nearly a third of the popular vote in Scotland, as it did in October 1974. It was hindered by the first-past-the-post system, because the Nationalist vote was fairly evenly spread across Scotland, unlike those of Labour and the Tories, which were generally bunched in specific constituencies, giving a much better chance of achieving success in some of them.

The additional member system in the Scottish Parliament, on the other hand, more accurately reflected the spread of a party’s support across the nation. This did not guarantee more SNP success, but in the long term made it more likely.

The Nationalists’ most stunning achievement came with the immolation of Labour in Scotland in the general election of May 2015. Some contend that this came about because Labour had to pay a terrible price for standing shoulder to shoulder with the Tories for the No campaign in the lead-up to the 2014 referendum, a rapprochement that many erstwhile supporters regarded as nothing less than a pact with the devil. That collaboration between old enemies might well have been a tipping point but, in my view, the fall of Scottish Labour had much deeper historical roots – and this might imply that the party’s recovery could also be a protracted process.

From the early 1960s until 2015, Labour relied in each general election on a haul of 40-plus House of Commons seats from Scotland in order to have a fighting chance of victory against the Conservatives, who for many decades have been buttressed by their serried ranks of seats in the south of England. Now, that Scottish sheet anchor has been set adrift.

In the 1990s, as a university teacher, I ­began to notice a steady and then increasing drift, among brighter undergraduates with a political interest, away from Labour, to which their parents and grandparents had given loyalty for so long, towards the SNP. That trend has predictably accelerated in recent years as issues of identity, devolution and then independence came into the Scottish political mainstream.

For many of Scotland’s youth now, nationalism is cool and Labour is old hat. A generation of potential leadership talent for the party is in the process of being lost and that quality will be irreplaceable.

Tom Devine’s “Independence or Union: Scotland’s Past and Scotland’s Present” is newly published by Allen Lane