How can Labour win back Scotland? That question has been an occasional, albeit muted theme of the Labour leadership contest, with each candidate offering their own credentials as the one who can reignite Labour’s Caledonian hearth. They tiptoe carefully around the cold and morbid pessimism that haunts the Scottish Labour party, chilled by another question that hovers unasked and unanswered over every party meeting: can Labour win back Scotland?

Answering that means understanding the causes of defeat. Though Labour was electorally dominant in Scotland until as recently as 2010, the party has in fact undergone the same long, secular decline over generations that has taken place more obviously in England. In the 1980s this was concealed by the broad, cross-class unpopularity of the Conservatives; anti-Tory votes gravitated naturally towards Labour, thanks to the binary choices encouraged by the first past the post electoral system. Scotland’s civic leaders claimed to reject Thatcherism, but the country did not escape the late-20th-century confluence of neoliberalism and postmodernity that carried away social housing and heavy industries as well as old community, familial and religious ties, eroding much of what had been Labour’s home turf.

Many in Scottish Labour struggle to understand why the SNP's failures have not already returned them to power

Amid this deeper structural decline, Labour was also able to sustain itself between the 1970s and 1990s by becoming the party of Scotland, combining with Scottish trade unions to fight for Scottish interests both in and against the Westminster state. They remade Scotland in Labour’s image, knitting together industry, faith and the public sector, combining manual workers and the white-collar world of “civic Scotland” in a coalition for the peaceful navigation of global change. “Labour Scotland” – as Gerry Hassan and Eric Shaw call it in their book The Strange Death of Labour Scotland – turned the lived experience of industrial Scotland into a powerful political identity that, for years, survived the loss of its natural habitat. One of its last achievements, but also perhaps its greatest, was the Scottish parliament.

Yet this strategic triumph set the scene for the party’s biggest unforced error. During the 2000s, instead of renewing its claim to be Scotland’s insurgent voice against Westminster, Labour – back in government north and south – allowed the SNP to occupy that terrain. By the time of the 2014 independence referendum, Labour had become Westminster’s voice in Scotland at a time when Scots – like many across the UK – were increasingly fed up with a distant and centralised British political system. Scottish Labour helped to keep the UK together, but its defence of the constitutional status quo sacrificed its credibility as being the best party to resist an unpopular Tory-Liberal government in Westminster. The results were brutal: according to the political scientists Ailsa Henderson and James Mitchell, of the Scots who voted Labour in 2010 and yes in 2014, an extraordinary 82% switched to the SNP in 2015 .

Are there any hopes for recovery? Fortunately for Scottish Labour, most Scots still vote for parties that position themselves on the centre left. Though the Scottish Conservatives are now the main opposition, the most likely alternative to the SNP for most SNP voters remains Labour. The SNP’s failures in government are increasingly hard to deny, and they are in precisely the areas of diligent service delivery – particularly education, transport and health – that Labour knows best.

Many in Scottish Labour struggle to understand why this hasn’t already returned them to power and blame “the constitutional divide” for distracting voters from their real interests. This is an excuse, not a strategy, based on a bitterly tribal refusal to wonder whether the voters might be on to something. Scottish Labour’s best electoral options clearly lie on the Scottish nationalist side of that “divide”, and there are copious resources in the Labour tradition for claiming back some of that ground.

Scottish Labour urges UK party to learn from fall of 'tartan wall' Read more

The SNP’s cosy relationship with multinational corporations such as Amazon and Airbnb, as well as the fossil fuels sector, also leaves miles of space for Scottish Labour to embrace an unambiguous economic nationalism based on democratic public ownership and state-driven investment in renewable industries.

The party already espouses this in policy terms but remains detached from the kind of visionary national project in which any policy programme needs to be embedded. The idea of an heroic, big-state “socialism” relies too heavily on a dated British left identity that supporters of independence have largely abandoned, while the proposed constitutional fix of “federalism” is too dependent on technocratic detail to offer the kind of populist clarity successful winning electoral coalitions are made of.

What is striking is how open the goal is, and how determinedly it is being ignored. Recent polls suggest that about 50% of Scots now support independence; it is likely that there are many more who want to but remain uncertain of Scotland’s ability to weather economic upheaval. The SNP’s “sustainable growth commission”, designed to assuage these fears, has already irritated the party’s left wing by arguing for disciplined deficit reduction and the continued use of sterling after independence.

Meanwhile, some in the SNP are suggesting that as long as the Conservatives refuse another independence referendum, they may have to wait for another Labour government before a vote is possible. The problem is that as long as Labour is dead set against both independence and a referendum, independence supporters can’t risk diminishing the SNP’s bargaining power should Labour get another shot at power.

Labour should support a new referendum on clear democratic grounds, backing Scotland’s right to revisit its place in the union with Brexit going ahead. But it can also take these opportunities to go a step further and pinch the SNP’s “gradualist” clothes. If Scotland’s economy is not quite ripe for self-government then only Scottish Labour can promise to use the full resources of the British state to create the best possible conditions for the independence in Europe that Scots increasingly want.

Scottish Labour could argue for continued union and a Labour government as part of a patient but more secure path to an independent Scotland, ensuring that some of the wealth sucked into London over the past few decades can be redistributed to build up the resilience and autonomy of the Scottish economy – as well as pursuing a relationship with Europe that reduces the potential for friction between a post-Brexit England and an independent Scotland in the EU. To recover, Scottish Labour needs a new and distinctly Scottish horizon that goes beyond mere “good governance” of public services and the hope that England will vote Labour. It just needs to listen to the voters it has lost.

• Rory Scothorne is completing a PhD on the radical left and Scottish nationalism between 1968 and 1992