In Scotland, the general election encapsulated Barack Obama’s Audacity of Hope: an expression of optimism, pride and national self-confidence that defied the cynicism (or realism) of the established order. In England, the opposite happened. Fear triumphed over hope: fear of “chaos”; fear of Ed Miliband’s socialism; fear of being held to ransom by the Scots. This fear was carefully – brilliantly – mobilised by the Conservatives and used to devastating effect in a targeted campaign that included 23 Tory-facing Lib Dem seats (all lost).

I know; I was a victim of it. My comfortable majority disappeared as thousands of suburban Londoners quietly feared for their (generally prosperous) existence. Fear is not anger. I have never been through an election (my ninth) and been greeted with, and misled by, so much personal goodwill and affection on the doorstep.

The fear was heard in growing volume through two seemingly innocuous words. The first was BUT: as in “We think you are doing a great job as our MP” and “We think your coalition is good for the country” BUT Miliband/Scotland/. . . “is a nightmare”.

The second word was TACTICAL. Tactical voting is normally embraced by Lib Dems in my part of the world as a signal to Labour and Green voters to “keep the Tories out”: our defence mechanism against the first-past-the-post system. After five years of Tory-Lib Dem coalition this message was more complicated, but, to my pleasant surprise, many voters heeded the advice of the Daily Mirror and Polly Toynbee and voted for me. Had that tactical vote remained at the 2010 level, I and a few more colleagues would have survived. But that wasn’t the central problem. For floating, uncertain voters the Tories had managed to redefine tactical voting: “We would like to vote for you as our MP locally but we have to vote tactically to keep out Labour nationally.”

One illusion buried by this political avalanche was the power of the brand “Popular Local Champion”. My own approval rate was plus 40 per cent – about mid-range for an incumbent Lib Dem – but this was not enough. I have no doubt that defeated but assiduous MPs north of the border, Lib Dem and Labour, feel much the same.

So where did these powerful emotional currents, fear and hope, come from? Why, in our well-ordered country, after five years of stable government following a “hung parliament”, was fear of “chaos” so potent? The immediate response is to blame Ed Miliband, who was successfully caricatured as someone trying to turn our country into an Anglo-Saxon version of East Germany. He clearly misread the public mood, which was fearful of change, or experiment, and distrustful of promises of a better life, especially when financed by the taxpayer. But he was as much a victim of circumstances as the creator of them. He was the product of a tribal Labour culture that had become severely disconnected from social and political realities.

The seeds of his failure were sown in the early days of the coalition government, before he was elected leader. I recall the howling, angry, self-righteous sea of Labour faces on the benches opposite. Furious at loss of office, bitter at the sense of betrayal (by Nick Clegg in particular) and without a shred of humility. I have always acknowledged that Gordon Brown and Alistair Darling made a good fist of managing the financial crisis and that the Blair-Brown years produced good, progressive change, especially at the beginning. But in 2010 there was a sense of entitlement to power, a belief that Ed Miliband reflected: that a capitalist crisis must inevitably require socialist solutions. Public finance was not understood, or even taken seriously; even as one of the lesser villains of the coalition, I was taunted for five years for making 25 per cent cuts in the departmental budget that my Labour predecessors had already inked in. The Tories’ skill was in crystallising the public distaste for Labour’s record and offer.

But if this alone were the source of English fear I would still be an MP with a comfortable majority. What none of us predicted was the Scottish effect. I wrote two Demos pamphlets about the “politics of identity”, the first of them two decades ago, speculating about how our politics could be moving from the old certainties of class and left-right debate to new divisions based on national identity, race, religion and language: Europe, immigration, Islam, Ireland – and Scotland. And in Twickenham, even though I often witness the passions of sporting nationalism, I never imagined that the Battles of Culloden and Bannockburn would be refought in the minds of my constituents. But in the event, the Scottish Problem carried a lot more weight than the bedroom tax or even the mansion tax.

The English reaction isn’t a racial thing. The prevalence of Scottish voices in our media reflects an underlying respect, liking and sense of trust. And Nicola Sturgeon’s appeal to many English people was not only to star-struck lefties, but also a positive reaction to an articulate, clever and attractive female politician. The greatest fear and loathing of Scottish nationalism I encountered came from Scottish expatriates settled in London. What they and many English voters resented was the idea of their country – Britain – being redefined without their consent, and without being consulted. To add insult to injury, Scotland seems to have much the better deal from the Barnett formula. The fear of a weak, Labour-led UK government being held to ransom by the SNP was just too much for a lot of my voters.

As I warned them, unsuccessfully, on the doorstep: be careful what you wish for. A Conservative UK government with minimal legitimacy in Scotland is just what the Nationalists want. Every failure and hardship north of the border will be explained away as the fault of the Tory Toffs in London. Attention can be deflected from those overdue, awkward questions about the chummy relationships with right-wing billionaires such as the Souters and Murdochs, or breathtakingly cynical policies such as paying for free university tuition by raiding the funds of further education colleges and schools (as well as English taxpayers).

***

Now that the Conservatives and the SNP have consummated their loveless marriage of convenience, they will have to learn to cohabit. The basis of the relationship is that the SNP cannot afford to lose a financially favourable settlement and David Cameron cannot afford to lose Scotland. It is possible this mutual self-interest will enable them to progress beyond the present top-down decentralisation of spending responsibilities to a genuinely federal arrangement – home rule. Scotland would be responsible for revenue-raising as well as spending and would continue to participate in a wide range of shared services, defence, foreign policy and common money. The parallel problem of “English votes on English issues” is more tractable and there are sensible, practical voting solutions already being touted in parliament.

What makes this scenario worryingly unpredictable, however, is that any new constitutional arrangements will no longer be the preserve of clever anoraks and bloodless public servants. Fear and resentment now lie not far below the surface. The politics of identity rests on raw emotion, not reason. Scotland could become like Ireland a century ago but without the bombs (hopefully). And what complicates the relationship further is that another issue of identity has to be tackled at the same time: Europe and the overlapping question of immigration.

Ukip had a good election, mostly at the expense of Labour and my party. The comical antics of the Ukip hierarchy and the lack of parliamentary representation can’t conceal the fact that Nigel Farage’s team is now in the Champions League of European Nationalists: not quite in the same class as Marine Le Pen’s outfit, but close. With their Conservative fellow-travellers they have now been gifted an open goal: to spend the next 18 months to two years championing the No campaign in the promised EU referendum. Those of us who warned of the unintended consequences of a referendum were right, but have lost; so the referendum will happen and will dominate our political life. The Yes campaign will, I assume, be led by the established political class: those who believe in the European project and those who believe we have to put up with it – Cameron, Clegg, Alex Salmond, Miliband’s successor, the CBI and the TUC – a less-than-happy band of brothers trying to sing from the same hymn sheet.

The chances of things going badly wrong are endless. Cameron’s negotiating dem­ands may not be met and his backbenchers will set impossible objectives in any event. Another, possibly terminal, Greek crisis or some other explosion in the eurozone will undermine the case for the status quo. The underlying sources of discontent will not and cannot be dealt with, because, as Farage correctly points out, freedom of movement (ie, immigration) is integral to the “four freedoms” of the single market; nor will the UK budgetary contribution be renegotiated again. The astute Mr Farage will have calculated already that he doesn’t need to win. An ideal outcome would be a 45 per cent losing vote, as in Scotland, providing a basis for perpetual English grievance and constant campaigning against the horrors of rule from Brussels.

The politics of fear may come back to haunt the Tories. It has unleashed English – alongside Scottish – nationalism. Ultimately this may prove more dangerous to them than the traditional enemies of Conservatism. They have started a fire and clever Lynton Crosby will no longer be around to advise them on how to put it out.

Whether the fire can be contained at all will depend in large measure on whether the Labour Party and the Liberal Democrats can recover and offer plausible alternatives. Both for now are in a very bad way politically, and it is possible that the Labour Party’s advantage over the Lib Dems is merely that it is bigger, and that it has more to lose and further to fall.

My own party, I hope, will progress soon from shock and gallows humour to rebuilding from the rubble. Our stock price is so low that it offers a buying opportunity and we have had a flood of 10,000 enthusiastic new members within days of defeat. My own team is back on the doorstep recruiting and is finding it difficult to find anyone who will own up to voting Conservative, though many claim to agree with us while looking at the floor. We already know that many of those who were frightened into voting Conservative are suffering buyers’ remorse, or soon will be, and will be less easily intimidated next time. We know that many of our basic values and messages have enduring value.

It is just possible that disillusionment with the Tories and with the nationalists in England and Scotland will set in so fast and go so deep that, as in the mid-1990s, there could be a pincer movement from the centre and centre left under plausible new leaders. Merely to state the hypothesis suggests, however, how far away it is. But to make it even possible, a lot has to happen, including our two parties deciding whether they are for ever locked in mortal tribal combat or, more sensibly, whether they are potential allies in a wider, progressive purpose of constitutional reform; a liberal approach to civil liberties; anti-nationalist and internationalist; and with a modern fusion of social democracy and market economics.

Vince Cable was the secretary of state for business, innovation and skills from 2010 to 2015 and was first elected as MP for Twickenham in 1997