Months after the SNP’s first Scottish parliament election win in 2007, George Foulkes, then a Labour MSP for the Lothians region, appeared on the BBC’s Scotland at Ten radio programme. Discussing the SNP’s early performance in government, Foulkes argued that the party were “on a very dangerous tack. What they are doing is trying to build up a situation in Scotland where the services are manifestly better than south of the border in a number of areas.” Colin Mackay, the show’s presenter, was confused. “Is that a bad thing?” “No,” admitted Foulkes, “but they are doing it deliberately...” The exchange has deservedly assumed a mythical status among nationalists, for it encapsulates something profound about Scottish Labour’s attitude not only to the SNP but to the whole idea of a distinctive, democratic Scottish politics.

The idea that policy might be a political battleground, rather than an issue of administrative competence, remains somewhat baffling to an old guard trained, in the decades before Holyrood arrived, to passively sweep up the votes of an understated nation. Overstated influence at Westminster and in local government allowed “Labour in Scotland” to hold Scottish democracy at arm’s length, often by the throat; the party’s response to electoral competition – especially from the SNP – has tended to be furiously tribal, and its determination to undermine Harold Wilson’s plans for devolution in the mid-1970s led to an extraordinary stitch-up being imposed on the Scottish party by the UK leadership. A special conference in 1974, at which Labour’s Scottish MPs were forced to submit to devolution through intense behind-the-scenes pressure, became known as the “Dalintober Street Massacre”.

This culture of high-stakes, white-hot internal conflict was a consequence of national dominance, but the opening-up of the Scottish party system has only intensified the internal struggle. Richard Leonard was elected as Scottish Labour leader in 2017 on a platform of breaking with a “managerial” approach to politics, and moving away from a “Better Together”-style pitch to Tory and Liberal Democrat unionists. He has attempted a radical appeal – arguing for state-driven, structural economic change – to the coalition of working class and left-wing middle class Scots who voted Yes in 2014 and for the SNP in 2015.

He has, however, faced two enormous obstacles: the first is the intransigent, vocal opposition of the party’s right wing – figures like Dumbarton MSP Jackie Baillie and Edinburgh South MP Ian Murray – whose local electoral interests and ideological conservatism are at odds with Leonard’s approach. This relates to the second, which is more serious: the people Leonard is trying to win round have not remotely forgiven Labour for its condescension and hostility towards pro-independence arguments during the 2014 referendum. Labour does not need to agree with those arguments to get a hearing from Yes voters, but it does need to demonstrate that it respects them – something Leonard is trying to do with new proposals for radical constitutional reform across the UK.

Despite this, his party has not experienced the kind of symbolic sweeping-clean of its political slate that Corbyn achieved down south. It is hard to overstate the disdain Yes voters feel for the image of Labour represented by people like Baillie and Murray, and the notion that they still exert an influence over the party is enough to make any prospective SNP switcher run a mile. Leonard has been less overtly insurgent than Corbyn, for much of the latter’s grassroots backing came from the kind of people who, in Scotland, have been turned off by Labour’s stance on independence. After initially trying to keep the peace, Leonard had to fire his main rivals – Baillie and Anas Sarwar – from the party’s “shadow cabinet” for briefing against him, avoiding the kind of ideological fireworks with which Corbynism announced its revolution in England.

This context has been almost entirely missing from media responses to John McDonnell’s declaration this week, during two events at the Edinburgh Festival, that he and Corbyn believe the Scottish parliament should be allowed to hold an independence referendum if it votes for one. The immediate response of many was to re-enact Dalintober Street, and conjure up an existential battle between the UK and Scottish parties as McDonnell plotted an alliance with the SNP MPs. Poor, hapless Leonard – a Corbyn ally, no less, and supposedly opposed to a second referendum – was thus cast as a sacrificial pawn in McDonnell’s cynical games, and every dull stereotype of both Scottish and UK Labour leaders could be happily confirmed.

In fact, the fallout from McDonnell’s remarks exposed the extent to which the Scottish and UK Labour leaderships are coming into alignment, however unevenly, on a new constitutional settlement for the whole UK – one that shows far more respect for regional and national democracy, and seeks a union based on popular, democratically expressed support for radical change. McDonnell doesn’t need to strike a deal with SNP MPs; he calculates that the SNP will not be willing to vote down Labour’s genuinely popular economic policies and risk letting the Tories back in.

Neither McDonnell nor Corbyn, both coming from backgrounds in extra-parliamentary politics, have particularly firm views on the constitution. But they do want to rebuild a voting base in Scotland that supports a degree of economic radicalism – that is, the kind of people who also tend to support independence. Channelling that desire for rupture into constitutional overhaul is, at worst, a necessary – and, to these old radicals, a rather exciting – price of trying to win those people round.

Leonard might have been caught out by McDonnell’s comments, but as I wrote last month, his recent trajectory has been towards a similar line of thought. It is not him but his opponents on the Scottish party’s right wing who are most antagonistic towards McDonnell. Yesterday Jackie Baillie, in her recently-acquired role as chair of the Scottish MSPs’ group, released a statement ostensibly on behalf of MSPs opposing McDonnell’s remarks, despite Leonard’s explicit instructions to avoid further confrontation when told about the statement.

Paul Sweeney, the Labour MP for Glasgow North East and shadow Scottish minister, has hinted at how Scottish Labour might try to fudge their way out of their crisis, maintaining some semblance of unionist obstructionism and carefully rationing any ground given to the SNP. He suggests that if the next Scottish parliamentary election in 2021 produces a majority for a second independence referendum, then Labour would back it on the condition that it includes a third option for a federal UK.

There is a lot wrong with this suggestion, practically, politically and in principle. The Scottish parliament already has a majority in favour of a second referendum: the SNP’s 2016 manifesto argued that a “material change” in Britain’s political circumstances – such as Brexit – would justify holding a second referendum; the Scottish Greens’ manifesto was more vague on the issue, but nevertheless declared that the party would back independence should another referendum take place.

Particularly determined unionists might deny that there’s a “mandate” there, but such a denial doesn’t exist in a vacuum. Going against Holyrood on the question means, essentially, siding with Westminster, and that’s hardly a good look for a party trying desperately to distinguish itself from a distant political establishment.

By deferring matters to 2021, however, Labour gives the same left-leaning Yes voters it wants to win round a new, cast-iron reason to vote SNP. Labour wants to indicate to these people that it’s got the message, and respects their democratic voice. Yet by claiming that it will only respect a pro-referendum majority in 2021, Scottish Labour is explicitly telling them to send the message again, and louder this time. This is already counterproductive enough. The “third option” condition suggests that once received, that message would be lost in translation as a Labour-led UK government invents a baffling new grammar of compromise: how would a vote for a federal UK state be legitimately binding, unless the referendum was UK-wide? And what mandate would Labour have to impose such conditions after 2021 without a Scottish majority? Sweeney’s suggestion solves few problems and creates several more.

All this qualification and bet-hedging results from a desire to solve the “problem” of Scottish nationhood, and find a final, one-line “answer” to the “national question.” Scottish Labour is used to administrative fixes — devolution was itself an attempt to contain an assertive new political nationalism within the existing structures of the British state. What the party needs to understand is that the popular appeal of Scottish nationalism has always been as a means of asserting some kind of democratic impulse — expressed, naturally, through a distinctive national politics — against the centralised, distant state power which Labour has often been too determined to defend.

Far from standing its ground, petrified with fear, against the prospect of Scottish democracy, Scottish Labour should be fighting to make that terrain its own. That means recognising at last that there is nothing democratic about using the megaphone of the state — and its English majority — to shout down Scotland’s demands.