Twenty years ago, Labour was in opposition and heading for certain victory in the coming election. Its modernising leader had routed the unions, reducing their power and reforming the party. He had won the trust of middle Britain, and had a radical plan for transforming the country. This was a historic moment for the party. Then, suddenly and shockingly, he died. I'm talking about John Smith, Labour's lost leader, the man who made the party electable.

History is cruel, and reputations are gained mainly by those who prevail, not those who create the necessary conditions for success. Few have the humility of Newton, who said: "If I have seen further, it is by standing on the shoulders of giants." Political parties need such generosity about their own past. The standard account of New Labour sees the election of Tony Blair as year zero. That's a myth – a dangerous and unnecessary one. Harold Wilson was right that the Labour party is a broad church. Remembering all those who created the past is the foundation for making a successful claim on the future.

So, Labour needs to reckon with John Smith. In many ways, 1994 is a foreign world. One of the most common descriptions of Smith was that he was like a bank manager. In those days, of course, banks were respectable institutions. The comparison denoted a steadiness, a caution: above all, a decency. That's a word people routinely used to describe Smith when he was in his prime – not just after his untimely death 20 years ago today. Now the closest we get to a term of approbation is "maverick". What a world we have lost.

Would Smith have won the 1997 general election? Undoubtedly. His one-member-one-vote reform laid the union bogeyman to rest. Black Wednesday shredded the economic credibility of the Tories, who then started their 20-year (and counting) war on Europe.

Would he have been a good prime minister? Certainly. He would have come into No 10 as an experienced cabinet minister who had learned his trade in the difficult days of the Callaghan government. The ethos would have been a very Scottish Christian – Presbyterian, even – socialism, but with that sparkle of wit and humour that Smith gave to everything. Blair and Brown would have shone, but in a different way – Tony as a fine, reforming home or education secretary and Gordon, without the torture of thwarted ambition, more able to use his undoubted talents fully. Neither would have succeeded Smith, though. The next prime minister would have been Harriet Harman.

Paul Keating once said to a hapless Australian opposition leader: "I want to do you slowly." Smith would never have been so blunt, but he would have had the same instinct about the Tories and Europe. He would have tortured them. A second victory would have come easily, with the third would have been earned by a refusal to join the Iraq war. Smith would have found no difficulty in echoing Wilson's refusal to send troops to Vietnam.

But where would Scotland stand? It would have its parliament – that is Smith's legacy. There wouldn't be a referendum now, as nationalism would have been contained. For one thing, Donald Dewar and Smith would have had a more collaborative relationship than Blair and Dewar. More importantly, Smith would have managed Scotland, and Scottish Labour politics, more actively. He would have made sure that the void left by Dewar's death was filled. If even a handful of the big Labour beasts from Scotland – from Cook, Robertson and Reid to Darling, Alexander and Murphy – had gone to the Scottish parliament, then Salmond could not have cut such a swath through Scottish politics.

In the end, though, these reflections need to be a spur to action. The public are frustrated by modern politics not because they think it doesn't matter, but because they know it does. The complaints about how politicians talk and act are, in the end, a demand for a different type of politics, one whose core value would be termed "decency". Working for that would be a fitting memorial for a man who had so much to give.