Over 150 years ago, in Ottawa, Illinois, Abraham Lincoln recognised something that is as true in politics today as it was then. Addressing the crowd in the first of seven debates against his opponent for the US senate, Lincoln observed:

“Public sentiment is everything. With public sentiment, nothing can fail; without it nothing can succeed. Consequently he who moulds public sentiment, goes deeper than he who enacts statutes or pronounces decisions.”

In the fevered pre-civil war era, as debates about slavery raged, Lincoln would have well recognised the need to identify the twists and turns in public sentiment and how to best steer it in his direction. But in those debates he was identifying something deeper: that public opinion can get away from you, and if you don’t keep up with it, it can overpower you.

And sometimes, it can also catch you unawares. Jeremy Corbyn would be first to admit that he didn’t expect to be leader of the Labour Party, but he tapped into an energy in the party and in the broader Labour movement that followed the general election.

Since May, lots of Labour Party members have been scratching their heads wondering how we ended up with the result we did. Everyone has their own theory, but the election delivered a result that John Curtice may have predicted, but left everyone who had been observing the campaign up close stunned. Armando Ianucci put it best in this year’s MacTaggart Lecture:

“It was the election in which the public punished the Lib Dems for not stopping the Tories and did so by voting in the Tories. It was the election in which the party advocating the Living Wage decisively lost, to a party now advocating the Living Wage. It was the one in which Nicola Sturgeon became the most popular hated politician in Britain… This place is a mess. We are a mess. We don’t know what we want.”

In Scotland over the past couple of years we in the Labour Party have been at the sharp end of public sentiment. Understanding what’s going on and how we respond to it is absolutely crucial to our recovery. And we have to start fairly simply: people hate politicians and parties with any connection to the past and a system they see as discredited, untrusted and corrupt. Despite what we all know – that the majority of men and women in politics are there with good intentions – public sentiment is not on our side and we need to win it back.

But we can’t do that by triangulating a response that ticks boxes with sectional interests here and there. We need to have a whole scale review of what we do, how we do it and who we stand for. I’ve written elsewhere about Justin Trudeau’s victory in Canada. It showed that even in a country where the deficit was as toxic as it is here, an argument about borrowing for investment could still win through as part of a real battle of ideas. Why? Because it was authentic and part of a broader appeal to people about the kind of country they want to live in.

It’s what Nicola Sturgeon knows and understands and why the Nationalists are still benefiting from the support they discovered during the referendum. It’s also what Jeremy Corbyn tapped into during his leadership campaign and why there is still an energy from his campaign months after his election.

It’s also what we need to take away from the Oldham by-election victory last week. Oldham showed that despite the clamour of the media and the predictions of near wipe out, Labour could still pull through, even increasing the share of our vote. For all the predictions and nay-saying, in the end a strong campaign with a credible and authentic local candidate and an energetic campaign on the ground won the day.

As we look ahead in Scotland, we are about to face the toughest Scottish Parliament elections we have ever had and two things are clear. First, public sentiment is still on the SNP’s side but, second, we have never faced a situation in which the public’s mood can be so volatile. In the post-referendum era, public opinion is still bitterly divided and the constitution dominates, but this election is an opportunity to start to talk about the things that we will be able to do with new powers over tax and welfare, and what the SNP haven’t done with the powers they have had since 2007. And that’s where our strength lies – ideas that can take the country forward.

But it can’t be used to hide the difficult work that has to be done – listening hard to people in communities across the country and responding to what they’re telling us. Ultimately, we can only progress as far as public support will allow us to and, when we have the support of only 1 in 4 of the electorate in Scotland, we need to shift opinion and embrace the fact that people are crying out for change and authenticity more than ever.

Labour under Jeremy Corbyn is the only party that aspires to truly unite our country – from the north of Scotland to the south of England. The Tories want to divide us by class and the SNP want to divide us by geography. If we get it right, that’s a powerful, hopeful message. It’s time for Labour to remember what it stands for, reflect the outrage that people still feel about the way politics used to be done and be a voice for working people the length and breadth of our country.

Martin McCluskey is standing for selection as a Scottish Labour Candidate for the West Scotland region for next year’s Scottish Parliament elections.