As we try to sift the meanings of the 2015 general election, it’s worth beginning with a fundamental but far too little-discussed problem for political journalism: how the hell do we know what we think we know? What value – if any – do commentators, set apart from the professional politicians, actually bring? It’s not surprising that most of the time we commentators don’t like to talk about this. This spring, we really must.

I pick up my information from four sources, I realise, all of them suspect. The first is the politicians; during this campaign I spoke regularly to the parties, and to old friends across the political spectrum. The party HQs proved to be either deluded, or lying: I was told again and again by Labour that its ground operation was superb and its numbers, particularly in the north-west of England and the Midlands (where the party was slaughtered) were very strong indeed. The Liberal Democrats assured me that Tory talk of destroying them in the English south-west was malicious and ridiculous. (Plainly, it wasn’t.) The Tories said they were doing fine, you know, fine-ish, but never sounded hugely confident.

So much for going to the top. More useful were politicians from all sides I’ve known for years. A series of experienced Labour people sounded pretty wobbly – I should have spent more time thinking about that and less about the messages from the centre. Some Conservative MPs, notably George Osborne, seemed much more confident than Cameron HQ.

The second source, and one that dominated everybody’s day-to-day thinking, was the polling. The pollsters didn’t get everything wrong: they picked up the huge Scottish story. Then again, anybody who stepped off the train at Edinburgh Waverley Station and bought a latte would have picked that up. But they were massively out on the main story. My deep frustration is that this tilted the whole conversation about politics – the reporting, and therefore the pressure the reporting placed on the politicians. To be specific, if we had known how close the Liberal Democrats were coming to wipeout, there might have been speculation about what that would mean. Everybody ignored this. Had we known how badly Labour was doing, there would have been much more pressure on Ed Miliband over his main economic message. There wasn’t nearly enough.

My third main source was the rest of the commentariat, this time including the self-appointed commentariat of Twitter. That’s a big range of voices. But during an election campaign, people retreat into their ideological bunkers. There are some who ask pene­trating questions, keep their heads tilted sceptically, and are worth following, but by and large journalists listening to other journalists only produces an echo chamber of lazy, received opinion, big on the volume, an ear-splitting background noise.

This leads me to the final source of information, always ridiculed and yet the one that proved most accurate and that I wish I had spent more time attending to – anecdote and random conversation. That is plainly dangerous: we are all prisoners of our own geographical and class location, however much we think we are in the swim. One of the advantages of having a televised face and jug ears, however, is that people come up to you the whole time and tell you, unprompted and unstoppably, what they think. As I was doing my daily walk to the shops, or sidling off to my local for a pint of IPA, I was buttonholed again and again.

What people wanted to say, in my part of London, could be broken down into two big themes. First, they hated the idea of a minority Labour government backed by the SNP. Almost immediately that this became a leading Tory theme, I was picking it up on the street. After the first two weeks of a Tory campaign focused on the economy generally and the uselessness of Ed Miliband, and which seemed to me to have been from their point of view wasted time, plainly the Conservatives had found something that was cutting through.

The second theme was that Labour apparently “hated” the self-employed, people running or working in small businesses, and anyone who’d had any kind of success. I’m going to come back to this, but it struck me at the time. A painter and decorator, for instance, who employs half a dozen others, walked across the street to say: “I can’t vote Labour. I work bloody hard. I’m the kind of person they despise . . .” As with the anti-SNP reaction, you ignore a single comment but when you hear the same kind of thing dozens of times, you know that something is going on.

***

Now this isn’t the complete sum total of what was going on in my head during the campaign. I’ve been covering these things since 1983, and echoes of John Major in 1992 reverberated. But note that, compared to his open and confrontational street oratory back then, in among the jeering, the leaders this time were sanitised and surrounded by pre-selected audiences. Jim Murphy, the Scottish Labour leader, put himself through it in Scotland; but apart from that, it seems to me that all the money spent by the broadcasters and newspapers on sending their correspondents to join Cameron, Miliband and Clegg on their bus tours was wasted. No election is just like any other. Having been around for a long time is no guarantee of wisdom.

Other sources will be more respected next time. The betting market wasn’t great – money was pouring on to Ed Miliband right at the last minute, by which time he’d already lost – but the gilt markets seemed to know what was going on. Number crunchers using historical voting data did pretty well. And intriguingly, those in the parties who had scanned postal voting returns were also in on the secret.

Finally, I’ve interviewed many dozens of politicians many times. I ought, therefore, to know where their weaknesses are – though that doesn’t necessarily translate into knowing what the public thinks. But if there is a logical problem obvious to me, presumably it is obvious to millions of ­other people. The ever more glazed and convoluted attempts by the two Eds to avoid saying that they had overspent while in office is a good example.

There was a perfectly rational way of dealing with this. They could have said: “Look, the overspending was relatively minor in historic terms and was supported by almost everybody at the time. And be very careful of describing the building of new hospitals, schools and nurseries as ‘profligate’ or ‘waste’: our alleged overspending has given Britain places where children are currently learning and their grandparents are having heart operations. It’s not like blowing too much money on your credit card in B&Q.

“At the time, none of us knew – not you, not the government, not David Cameron or George Osborne – that an obscure housing crisis in Middle America was going to bring down the entire banking system.”

They could have said that. They didn’t.

Why not? Because, had they engaged in the conversation seriously, they would have had to go on to say something like: “However, given what we know now, do we wish we had spent a little less during the good times? Of course we do.” And, true or not, they thought that the Conservative media would have translated this into: “Ed admits, ‘Yes, the crash was our fault.’”

That seems to me to have been a huge tactical mistake on the Labour side. The party should have engaged in the argument, and made its case, not unreasonable, long before the proper election campaign started. As I have written here before, by allowing the Conservatives to set that narrative they handed David Cameron a huge weapon, which he used during April and May almost every day.

There were big mistakes on the Tory side, too, I thought. I was flabbergasted when Cameron and Osborne suddenly found a minimum of £8bn, and perhaps a lot more, for the NHS right at the last minute. It seemed to me to blow a hole in their “uncosted spending commitments” attack on Labour. In the end, it didn’t seem to matter, because voters presumably thought that ­Labour would always spend and borrow more than the Tories, whatever anybody said. Still, it was a heck of a risk.

***

But the biggest issue that emerged during interviews was, as I found on the street, the SNP surge. I have been studying and writing about the Scottish National Party since I was in my twenties. I wish I had been back in Scotland much more, and earlier, but I’ve been up there quite a lot recently, working and visiting family. So I know that the SNP is not the Maoist threat Middle England cowers from. Among its new MPs are former Conservatives, business people, all types. It’s true that its new 110,000 membership contains many on the hard left who have made a lot of noise, but the party is a complex phenomenon.

Still, the “watch my lips, no deals” rebuttals by Ed Miliband were always going to be difficult. He might have ruled out a coalition and a formal confidence-and-supply arrangement, but the numbers dictated that a minority Labour government – the best he could hope for – would have to rely on SNP acquiescence at best week by week.

Now, I have no hard evidence for what follows, but I don’t believe that English ­voters’ hostility to an SNP-influenced outcome was anti-Scottish. I think it was the combination of the thought of a relatively weak government, which would have to negotiate its way through its programme, with the anti-Trident and anti-austerity messages of the SNP, that spooked much of Middle England.

Don’t forget that all the voters who do not want Trident, and were against austerity, weren’t up for grabs anyway: they were ­already committed, presumably, to Labour or the Green Party. So a Scottish National Party programme, crafted to appeal to Glasgow Govan and Dundee, didn’t go down so well with soft Tory voters and Liberal Democrats in Cirencester or the Peak District. Quelle surprise!

That, for me, was the story of this election campaign. Looking ahead, what lessons can we draw from it?

Living in London, I am very cautious about trying to predict what is going to happen in Scotland next. But the following things seem almost self-evident. First, to have any chance of revival, Scottish Labour has to separate itself from the party south of the border. It has to begin again, unionist but with its own head office; back to the party’s origins. This may not work, either. Second, in politics, winning brings new problems and winning big brings bigger new problems. Nicola Sturgeon now not only has to negotiate with somebody she never thought she’d have to deal with, and who has the authority of a big election win, she has to manage the enormous expectations of what looked like, in effect, a Scottish revolution. I couldn’t fit into her shoes in a million years; but I’m glad I don’t have to try.

As to the shattered English left, I go back first to all those conversations about Labour not being for “people like me”. Labour politics works when it is, in effect, an alliance between the bulk of people working in private companies, big and small, and those on the margins. Now, if Labour chooses to forget about people on benefits, those on poverty wages, and the huge inequities caused by a super-rich global class, then it ceases to have any purpose. But it simply can’t get itself into a position again where shopkeepers, tradesmen and all those who want to better themselves think Labour “hates” them.

I don’t suppose this was something that Ed Miliband or those around him ever set out to achieve. It was more about tone, and where they came from, and what their own instincts were. Britain is brimming with relatively affluent (or at least comfortable) non-socialists who have a strong sense of community and social altruism. They support homelessness projects run by churches, they back local campaigns, they spend spare income not on bigger cars but on Oxfam appeals. They are good people. They just happen to be outside the immediate reach of the state. Labour can sometimes give the impression that only those working in the public sector or those on benefits are virtuous and admirable. This is politically lethal.

***

Is that more important, or less important, than confronting the “Blue Labour” questions of immigration, low pay and embattled trade unionism? I don’t know – but reaching out to the majority isn’t a luxury.

So, Labour has a cultural problem to resolve. It’s about how the party speaks, the way it pitches its appeal. It is vastly more important than who the next leader is. Over the next few years, we will see, I suspect, little real sign of a Labour advance in Scotland – the defeat is so profound that it will take many years to recover – while in England boundary changes further entrench the Conservatives. Unless Labour has the courage and imagination to reform itself completely, it has no chance of recovery.

Democracy is a pendulum. Not even the biggest and most unexpected victories last; in fact, they contain the seeds of the next defeat. Yet this assumes and requires that the defeated parties learn hard lessons. The Tories’ biggest problem ahead isn’t the EU referendum. It isn’t even the relationship between London and Scotland – federal Britain is taking shape before our eyes. No, it is their relationship with big money, the global financial system that remains unstable and often incompetent. David Cameron is a more sophisticated and flexible Tory leader than many understand. But I don’t see him trying to fix that problem, and that leaves Conservatism vulnerable.

Ought Labour and the Liberal Democrats to forget their differences and try to merge? Probably not: they have different philosophies and those differences matter. But it would be a good, useful and salutary thing for both of them to contemplate the possibility. A big election defeat ought to shatter old ways of thinking. It’s important not to waste a good defeat. I have spent the past few days doing two things – sleeping and worrying about how I do my job.

Defeated politicians, as well as humbled journalists, could do worse.

Andrew Marr’s most recent book is his novel, “Head of State” (Fourth Estate)