As poll after poll suggested that the Scottish National party was set to make historic gains on 7 May, several outgoing Labour MPs found solace in gallows humour. “I’m now set to Defcon fucked,” remarked one. “It doesn’t matter how good you are or how weak your [SNP] opponent is – it’s over.” Another said the mood in Scotland resembled “the last days of Rome”, just “without sex, or wine. In fact, with none of the fun bits.” Others simply spoke of an oncoming “tsunami”.

On the night itself, only three unionist MPs found themselves on high enough ground when that nationalist wave hit. The electoral map of Scotland had become a sea of yellow, with just a speck of blue in the south, red in the east and orange in the north. Scotland’s once-dominant centre-left party, observed the then Scottish Labour leader, Jim Murphy, had been “overwhelmed by history and circumstance”.

A hundred and fifty miles north-east of Glasgow, the former first minister, Alex Salmond, himself one of the 56 new SNP MPs, had another explanation. The “Scottish lion”, he declared, had “roared”, producing a swing “the like of which [had] not been seen in recorded politics”. That much was self‑evident, but how in the space of just five years had the SNP gone from 6 to 56 MPs, from 20% to 50% of the vote?

Much of the instant analysis was inevitably reductive. Labour, claimed some, had not been adequately leftwing in a country where the political centre of gravity (or so it appeared) had long been at odds with that in England. Others lamented the party standing “shoulder to shoulder” with the Conservatives to keep Scotland in the union the previous year, suggesting that Labour had suffered guilt by association. More generally, commentators and activists grabbed hold of any explanation that came to mind: Trident, austerity, even Tony Blair.

But in 21st-century Scotland – just as in the rest of the UK – labels such as “left” and “right”, and even “unionist” and “nationalist”, had long ceased to be useful. There was a tendency to depict Scotland as somehow firmly unionist until a decade or so ago, and avowedly nationalist since. In reality it had been both at the same time – and, indeed, had always been so – it was merely a question of degree. The fusion of Scotland with England in 1707 created not only a multinational state but a constitutional palimpsest, a nation both Scottish and British, under which all Scots could not help but possess a dual identity, simultaneously nationalist and unionist.

Over time it became obvious that political success rested upon managing the balance between those two identities, between the institutional tenets of Scottish identity – religion, education and law – and a broader British state that derived its strength from empire, protestantism and defence. So when, in the wake of this year’s general election, Nicola Sturgeon promised to “stand up for Scotland”, she was merely pursuing much the same strategy as her predecessors: first ministers, secretaries of state for Scotland, lord advocates – unionists, devolutionists and nationalists, stretching back more than 300 years.

Facebook Twitter Pinterest An SNP delegate reads the Scotsman on his way to the party’s special conference in Perth in 1998. Photograph: Murdo Macleod

Scottish nationalism had long spoken with different voices – literary, cultural, sporting and political. But those voices also needed to tell a story, a story of Scottish identity and the nation’s place not only within the UK but in a changing world. Thus, whoever had control of Scotland’s story most likely controlled its political discourse and therefore, but not always, its levers of power. During the late 19th and early 20th centuries power had changed hands many times, from Liberal home-rulers in the age of Gladstone and Lloyd George, to patriotic Scottish unionists in the age of Churchill and Macmillan. And, until around 2010, the Labour party had told a story of Red Clydeside, the welfare state and solidarity with the working classes of England, Wales and Northern Ireland.

But in the last year or so, as one outgoing Scottish Labour MP observed in the wake of election defeat, his party found itself fighting an alternative narrative – that of the SNP – rather than “a set of policies”. The beguiling narrative that the SNP offered focused on the great things an independent – or fiscally autonomous – Scotland might be able to achieve.

Wise leaders always tell a story about the country they aspire to lead, even if these stories are inevitably a mixture of fact and fiction. Just as Stanley Baldwin and John Major conjured an idealised England of pastoral scenes and cricket, from around 2004 Salmond and Nicola Sturgeon began to paint a picture of a leftwing Scotland, an anti-Tory Scotland, an egalitarian and outward-looking nation; an image of how many Scots liked to see themselves, if not a wholly accurate representation of who they actually were.

* * *

In the decades preceding the late 1990s, Labour dominated Scottish politics, both in local government and at Westminster. Since its founding in 1934, the SNP had experienced its share of surges – the first in the late 1960s, another in the mid-1970s – but more often than not, the party found itself on the political sidelines, its claim to speak for Scotland ignored or disbelieved by most voters. Even as recently as the early 2000s, many thought that devolution had killed off Scottish nationalism for good.

A number of key elements lie behind the SNP’s remarkable ascendancy since 2007, including the strategy of political triangulation (the art of stealing your opponent’s ideas), the decline of the Scottish Labour party and, above all, Alex Salmond’s masterful construction of an overarching story of modern Scotland. But the first major step towards rejuvenating the SNP came in 2000, when Salmond persuaded his party to adopt a new path towards independence. Before then, nationalists had argued that a majority of Scottish seats in either the House of Commons or a devolved Scottish parliament would constitute a mandate for independence negotiations. There was no need, they argued, for a referendum. But Salmond and others on the “gradualist” wing of the party realised this scared certain voters; what was needed was something to separate the SNP from its longstanding aim.

Salmond’s first big step towards rejuvenating the SNP came in 2000, when he persuaded his party to adopt a new path

Following a close vote, “independence must be achieved by a referendum” became the new SNP orthodoxy. This shift in policy was not welcomed by more fundamentalist nationalists. To them, the word “referendum” generally meant a unionist trap, the sort used to thwart Scotland’s constitutional ambitions rather than advance them. That, to be fair, had been the case in 1979 when a majority of Scots had backed a Scottish assembly only to fall foul of the “40% rule”, under which two-fifths of the entire electorate (rather than those voting) had to endorse devolution for it to take effect. Even in 1996, when Tony Blair announced that the creation of a Scottish parliament would depend on approval in a pre-legislative referendum, some in the SNP suspected that Labour was reverting to unionist type. (Although, in the end, the 1997 referendum ended up aiding the devolution process by imbuing it with a popular mandate.)

By declaring that a referendum was the route to independence, the SNP had taken its first significant step towards reversing its long-marginal electoral fortunes. But it did not look that way at the time. In July 2000, Salmond quit as SNP leader, believing that the media had tired of him as party leader. For four unhappy years his successor John Swinney worked hard but lost support, as the party fought among itself over ideological direction and electoral strategy. After losing ground at the 2003 Scottish parliament election and in a European parliament contest the following year, Swinney fell on his sword. The stage was set for Salmond to make a surprise comeback, with Nicola Sturgeon – who had withdrawn her own candidature – as his running mate.

The return of Salmond was the second key element in the changing fortunes of the SNP, for he was a master of appealing to voters across the political spectrum by adopting an apparently pragmatic mixture of rightwing and leftwing policies. By 2003, the 50-year-old Salmond had moved even farther from his youthful flirtation with Marxism. Always more interested in tactics than policy, at the end of the 1980s he had looked across the Irish Sea and liked what he saw: a Celtic Tiger slashing corporation tax and attracting international businesses to Dublin. If Ireland could do it, why not an independent Scotland? Interviewed in 1991, just months after Margaret Thatcher’s downfall, Salmond claimed “independence in Europe” could slash interest rates, while making clear his commitment to the marketplace and a “competitive and productive” Scottish economy.

Facebook Twitter Pinterest Alex Salmond in 1997. Photograph: Murdo Macleod

In this regard, Salmond was Blairite before Blair. During his first term as leader, from 1990 to 2000, Salmond jettisoned nationalisation, along with higher income tax rates. Unlike Blair, however, he took care to mask all the resulting policy shifts with a heavy dose of social-democratic rhetoric. In doing so, the SNP gradually became all things to all men – and much more effectively than the Liberal Democrats had ever managed. The party courted the business community with a liberal fiscal regime (Salmond promised less regulatory red tape), while Scotland’s public sector was promised protection from privatisation and falling wages. Thus the SNP, like Blair, built support in the centre ground and among Scotland’s Asian and Catholic communities – both tough nuts to crack for a party long perceived as white and protestant.

* * *

A year into Alex Salmond’s second term as leader, meanwhile, the party took another significant step. Until then, the nationalist narrative had largely been a negative one, infused with grievance (perceived and real), and much clearer about what it opposed rather than what it was for. But in 2005, SNP strategists identified five key priorities: “communications, governance, message, organisation and resource”. Detailed policy or ideology were not among them, but in presentational terms a new, relentlessly positive campaigning machine was about to be born.

Of course policy did play a part, but it was couched in ambiguous terms. The working title of the SNP’s 2007 manifesto, for example, was A Culture of Independence. The previous year Sturgeon had conceded in an interview that as Labour shifted to the right, the SNP had, in turn, perhaps become rather “centralist and statist” in its outlook. Now it was time for a change.

Sturgeon, who would later do more than anyone to make Scottish nationalism synonymous with a centre-left agenda, sought to depict independence as both constitutional and personal. Governments, she said, “should not go around getting in people’s way when they don’t need to”, a remark that would not have sounded out of place coming from a small-state Conservative. Getting rid of student debt, lightening the burden of local taxation and encouraging small businesses, argued Sturgeon in 2006, was all intended “to give individuals a fair crack of the whip … aspirational policies about people progressing and making the most of themselves”. Such an agenda, hoped the SNP, would “strike a chord with people in the middle classes”. Given how much of a battleground the aspirational agenda would later become, wielded against Labour as proof of its capitulation to Conservative rhetoric, this was significant.

This was social democracy, but certainly not the variety familiar to the Scandinavian countries Sturgeon et al had made a show of citing over the past few years. Furthermore, the SNP’s ideological position did not differ significantly from that of New Labour, which it had depicted as little more than a pale imitation of the previous Conservative governments in terms of its capitulation to a free-market agenda. Income tax was a case in point; although the SNP had fought the 1999 Scottish parliament election on the basis of raising the basic rate of income tax (which the new legislature could vary up or down by 3p), thereafter Salmond had abandoned a higher-tax agenda that might have given credence to his desire to emulate Sweden or Norway. Instead he preferred to pursue the political alchemy of a low-tax, high‑spend economic model.

It underlined that although the SNP had long dreamed of displacing Labour as Scotland’s dominant centre-left party, any prospect of doing so with a genuinely radical policy agenda had been shelved in the 1980s. Nonetheless, there remained important points of differentiation: tuition fees, for example, became the leitmotif of the nationalist approach – an explicit rejection of the Blairite education agenda. A longstanding commitment to unilateral nuclear disarmament also enabled nationalists to present themselves as more leftwing than Labour, and while on Trident they clearly were, it conveniently overshadowed similarities in economic terms.

Facebook Twitter Pinterest Nicola Sturgeon campaigning for the Glasgow Govan seat in the 1997 general election. Photograph: Murdo Macleod

* * *

The decline of the Scottish Labour party was the third decisive factor in the rise of the SNP. In 2006 a YouGov poll found that 56% of Scots agreed that Labour had “been in power too long in Scotland” and that it was “time for a change”. That sentiment, and more specifically the phrase “it’s time”, would form the basis of the SNP’s 2007 Holyrood election campaign. To drive home the message, Salmond depicted Labour as incapable of functioning “without remote control from their London masters”, cleverly contrasting his political autonomy with that of Labour’s Jack McConnell, Scotland’s first minister since 2001.

Salmond benefited from a perfect storm: continuing fallout from the 2003 invasion of Iraq, Liberal Democrat turmoil (Charles Kennedy had been ousted as leader) and an ongoing battle between Blair and Brown. This was the point at which Scottish Labour lost control of Scotland’s “story”. “We started to feel as if we were swimming against a tide,” reflected McConnell a few years later. “During the 2007 campaign [the SNP] picked up on the fact that Scots wanted to hear positive stuff so they ran with a very positive campaign and we ended up on the other side of that, sounding negative.”

That positive campaign got them across the finishing line – just – beating Labour to become Scotland’s largest party by a single seat at the 2007 Holyrood election. Salmond quickly presented his party as the rightful winner and, when coalition negotiations with the Liberal Democrats failed over the question of an independence referendum, he formed a minority government. As Tony Blair recounted in his memoirs, he realised that once Salmond had “his feet under the table”, he would “play off against the Westminster government and embed himself”. “It would be far harder,” predicted Blair, “to remove him than to stop him in the first place.”

Even from that precarious position, as Blair had anticipated, the new first minister made like Teddy Roosevelt, using his official residence, Bute House, as a bully pulpit to build support for the SNP and independence. The Scottish executive was rebranded as the Scottish government, while Salmond made a point of looking and acting like he already led an independent nation; even the early release of the Lockerbie bomber played to his story of a modern, compassionate nation, while signalling that from then on Scotland would be master of its own affairs, even if that upset Westminster.

Its performance in government was more competent than radical. Despite plentiful rhetoric about reducing inequality and promoting social justice, the SNP would not produce a record to match that of Labour between 1997 and 2010, which had reduced pensioner and child poverty. Yet what resonated was the story told over the next four years, of a Scottish parliament that had finally found its voice. As even Tony Blair’s former spin doctor Alastair Campbell conceded in a newspaper article, Salmond’s efforts after 2007 to capture the mood of Scots in terms of values and vision had provided “a clever soundtrack to … an upbeat, devolved Scotland”.

Facebook Twitter Pinterest The final day of the Scottish referendum campaign in September last year. Photograph: Murdo MacLeod

In the 2011 Scottish parliament election, the SNP performed astonishingly well, winning an overall majority in an electoral system specifically designed to prevent such an outcome. Labour had failed to respond to defeat four years previously. But there was a wider problem. By 2011, Unionism – be it red, yellow or blue – had become thoroughly banal, stale and defensive. Narratives of the welfare state and Britain’s influential place in the world had been abandoned – now it relied on balance sheets to make its case.

But, paradoxically, as Scotland’s voting behaviour continued to diverge from the rest of the UK, it increasingly resembled the rest of the country. Even in the midst of a recession, Scotland was now prosperous, a far cry from its position for much of the postwar era. On social issues, too, it had changed significantly. When it came to homosexuality, divorce and abortion, Scotland had thrown off its Calvinist roots and become as liberal as England and Wales. Survey after survey also revealed that political beliefs had, if anything, shifted rightward; cool on immigration, cynical about benefits and, like the thrifty Scots Margaret Thatcher always believed them to be, keen to withhold as much of their income from the state as possible.

Nevertheless, during the long referendum campaign that followed the SNP’s 2011 election victory, Labour not only lost control of the political narrative but increasingly lost permission even to be heard by growing numbers of once-loyal voters. The SNP said that Labour had lost its soul and ditched its principles, almost as if independence was somehow going to restore common ownership of “the means of production, distribution and exchange”, as stated in the 1918 Labour party constitution. Labour hit back on points of detail – often good ones – but ended up looking, especially as it campaigned for a “No” vote alongside Conservatives, as if it no longer believed its own story, that Scotland was more leftwing, more egalitarian, anti-Tory.

Labour not only lost control of the political narrative but increasingly lost permission even to be heard

Of course there were contradictions within the competing nationalist narrative: it was anti-nuclear but pro-Nato, committed to free university tuition but content to cut college places. From the comfort of the moral high ground the SNP railed against the Liberal Democrats for U-turning on tuition fees, conveniently forgetting that its own 2007 pledge to write off student debt had never been implemented. Nationalists were pro-EU but opposed to the single currency, common fisheries policy and increased centralisation; economically it talked orthodox left but intended to act conventionally right, pledging to cut corporation tax, and keep personal taxation low.

The SNP also cleverly co-opted aspects of the union that retained an emotional appeal; Salmond spoke of Scotland’s “six unions” – political (Westminster), monetary (sterling), regal (monarchy), defence (Nato), European (the EU) and social (family ties) – and pledged to preserve all but the first. This, as the journalist Alf Young observed, made the SNP leader “five sixths a unionist”, but the story he told was large and could therefore contain multitudes.

History also played its part, for Scottish nationalism was not unusual in embracing both “ethnic” and “civic” elements. In June 2014, the Scottish government marked the 700th anniversary of Bannockburn with a historical re-enactment of Scotland’s first war of independence. Even referendum day was imbued with historical significance, taking place exactly 100 years after the ill-fated Government of Ireland Act had received royal assent on 18 September 1914, the British state’s final attempt to appease Ireland on the eve of the first world war. History might not be repeating itself – although there were many echoes of the Irish debate in Scotland – but the symbolism was striking.

* * *

For the past century all of Scotland’s political parties have played the game – to varying degrees – of defining Scotland against Westminster, each of them claiming to “stand up” for distinct Scottish characteristics and interests. From the end of the 19th century onwards, the old Liberal party harnessed Scottish nationalism for nearly three decades, telling a progressive story of land reform and a nascent welfare state; then the Conservatives encouraged what they called “administrative devolution”, the steady transfer of Scottish Office staff and responsibilities from London to Edinburgh, as did Labour with its devolution rallying cry throughout the 1980s and 1990s.

Each had become a victim of their own success. Having delivered home rule for Ireland in the early 1920s, the Liberals began their strange death, and thereafter, having long framed Scotland as a “state within a state”, Conservatives suffered as voters pushed that to its logical conclusion by supporting legislative as well as administrative devolution. During the Thatcher years, Scottish Labour deployed nationalist weaponry against Conservative governments and, in doing so, unwittingly helped delegitimise aspects of the British state. More recently, having delivered a Scottish parliament in 1999, Labour lost its unique selling point to Scottish voters, and so another “strange death”, that of Labour Scotland, mirrored that of the Liberals 90 years before.

Given that historical context, the fact that independence was not endorsed by a majority of voters on 18 September 2014 was less significant than it initially appeared. As Iain Macwhirter, the political commentator for The Herald, put it in his book Disunited Kingdom: “Unionists didn’t quite win, and the yes campaign didn’t quite lose”. Two years of discussion and debate about independence had served to normalise the concept, which, against a backdrop of austerity and distant Tory government, appeared less risky than it had even in the oil-rich 1980s.

Alex Salmond clearly regarded the 45% yes vote as more of a lost battle than a lost war; he spoke of Scotland’s “100-year home rule journey”, skirting over the fact that much of the journey had concerned devolution rather than independence, more autonomy within the UK rather than full independence outside it. But shrewdly, he had taken up the old battle cries associated with the long campaign for a Scottish parliament, selling independence as a means by which to promote economic growth, social democracy and the slippery concept of Scottish values.

This narrative had become so entrenched, so appealing even to those who did not consider themselves nationalists, that not long after a Labour government dominated by two Scottish MPs, Gordon Brown and Alistair Darling, the SNP was able to argue that not only was Labour anti-Scottish but that Scotland lacked clout at Westminster. The traction that this narrative gained was evidence that the victors do not always write the first draft of history. The SNP line was reinforced through skilful rhetoric and soundbites, embellished and repeated until questioning it became almost pointless.

The SNP was able to argue that not only was Labour anti-Scottish but that Scotland lacked clout at Westminster

Salmond had always seen the bigger picture, possessing an ability to map out future political terrain and act accordingly. And having done so on 19 September he instinctively knew that a swift resignation would aid the SNP in the eight months between the referendum and the 2015 general election. The future, he explained, was “redolent with possibility”, and it would fall to Nicola Sturgeon – a less divisive and in many ways more talented politician – to take the party and broader independence movement to the next level.

The SNP, not for the first time, proved lucky in terms of external events and its enemies. Having long struggled to appear relevant in the context of a Westminster election, the near-certainty of another hung parliament made the image of a large group of nationalist MPs as powerbrokers appear credible for the first time since the 1970s. At the same time, when Scottish Labour leader Johann Lamont resigned in October 2014, claiming that the UK Labour leadership regarded Scotland as a “branch office”, nationalists could feel vindicated.

By the close of 2014, the SNP found itself in complete command of Scotland’s political narrative. Opinion polls indicated that most of the 45% who had supported independence now intended to vote SNP on 7 May. With Salmond and Sturgeon liked and trusted by a majority (even non-SNP voters) of the electorate, they were able to set the terms of debate in the months leading up to polling day, chiefly that the Scottish Labour party had “lost its soul”, that the cross-party Smith commission on more powers for the Scottish parliament was destined to fail, and only the SNP could possibly stand up for Scotland at Westminster.

For those who looked closely at the SNP and Scottish Labour manifestos, however, beyond Trident the pledges and philosophy therein appeared virtually identical, Sturgeon having belatedly followed Labour’s lead (rather than the other way round) on a mansion tax, bankers’ bonus and abolition of non-dom status. But in politics perception is everything, and voters, including a huge tranche of Labour supporters who had defied their party by voting yes in the referendum, believed the SNP leader to be more authentically leftwing than Scottish Labour’s Jim Murphy. The two leaders, and their parties, were judged by completely different standards; thus Labour’s brand had failed, just like that of the Tories in 1997. Sturgeon, meanwhile, jettisoned the neoliberal elements of her predecessor’s agenda and spoke of marrying social justice and economic growth as if it were a bold new departure in political economy.

On 7 May, the SNP won 50% of the vote. Once distorted by first-past-the-post and up against a unionist vote split three ways, it was enough to produce a landslide. Not only did that result kill the notion of British party politics, it demonstrated that in Scotland “nation” (and its associated identity) had become the dominant axis upon which discourse, public policy and even crude party politics turned. Unionists might try to recalibrate that reality by highlighting points of detail, but for nearly a decade that had been subject to the law of diminishing returns. And for much longer than that, the substance of Scotland’s exceptionalism had not been what mattered in political terms, rather a sense of being different and therefore a desire to do things – often the same things – for itself.

Did the SNP - unlike its predecessors - possess the answers to crucial questions about inequality, economic growth and sustainability? Did Alex Salmond and Nicola Sturgeon have original, radical policy solutions that could be put into practice? By the time 7 May arrived, these were no longer the questions that mattered. For, in the context of a United Kingdom in flux, a troubled Europe and an ever-turbulent world, they were beyond any shadow of a doubt the best storytellers.

• Follow the Guardian Long Read on Twitter: @gdnlongread