Politics for a progressive has been gloomy for a long time. On Thursday, in Richmond Park of all places, there was a ray of light. Progressive parties (at least some of them) and ordinary voters combined to beat Ukip, the Tories and their "hard Brexit, soft racist" candidate.

It didn’t happen by accident. Let's be clear, the Liberal Democrats do by-elections really well. Their activists flood in, and good luck to them. But Richmond Park was too big a mountain for even their focused efforts. No, the narrow win was also down to the fast growing idea of a progressive alliance.

The progressive alliance is both a defensive and offensive move. It recognises the tactical weakness of progressives under first past the post – a system the Tories and their press know how to game. With progressive forces spilt between Labour, Liberal Democrats, Greens, the SNP, Plaid Cymru, the Women’s Equality Party and more – there is no choice but to co-operate, bring in proportional representation and then a whole new political world begins.

This move opens up the wider strategy – to end the domination of the City, and right-wing newspapers like the Mail, so Britain can have a real debate and make real choices about what sort of economy and society it wants. A pipedream? Well, maybe. But last night the fuse was lit in Richmond Park. The progressive alliance can work.

Months before the by-election, the pressure group for a progressive alliance that I chair, Compass, the Greens, and some Labour, Liberal Democrat and SNP MPs and activists, began considering this. The alternative after Brexit was staring into the void.

Then the Tory MP Zac Goldsmith stepped down over Heathrow. To be fair, he had pledged to do this, and we should have been better prepared. In the event, urgent behind-the-scenes calls were made between the Greens and the Liberal Democrats. Compass acted as the safe house. The Greens, wonderfully, clung onto democracy – the local party had to decide. And they decided to stand up for a new politics. Andree Frieze would have been the Green candidate, and enjoyed her moment in the autumn sun. She and her party turned it down for a greater good. So did the Women’s Equality Party.

Meanwhile, what about Labour? Last time, they came a distant third. Again the phones were hit and meetings held. There was growing support not to stand. But what would they get back from the Liberal Democrats, and what did the rules say about not standing? It was getting close to the wire. I spent an hour after midnight, in the freezing cold of Aberdeen, on the phone to a sympathetic Labour MP trying to work out what the party rule book said before the selection meeting.

At the meeting, I am told, a move was made from the floor not to select. The London regional official ruled it out of order and said a candidate would be imposed if they didn’t select. Some members walked out at this point. Where was the new kinder, gentler politics? Where was membership democracy? Fast forward to last night, and the Labour candidate got less votes than the party has members.

The idea of a progressive alliance in Richmond was then cemented in a draughty church hall on the first Tuesday of the campaign – the Unitarian Church of course. Within 48 hours notice, 200 local activist of all parties and none had come together to hear the case for a progressive alliance. Both the Greens and Compass produced literature to make the case for voting for the best-placed progressive candidate. The Liberal Democrats wove their by-election magic. And together we won.

It’s a small victory – but it shows what is possible. Labour is going to have to think very hard whether it wants to stay outside of this, when so many MPs and members see it as common sense. The lurch to the right has to be stopped – a progressive alliance, in which Labour is the biggest tent in the campsite, is the only hope.

In the New Year, the Progressive Alliance will be officially launched with a steering committee, website and activists tool-kit. There will also be a trained by-election hit squad, manifestos of ideas and alliances build locally and across civil society.

There are lots of problems that lie ahead - Labour tribalism, the 52 per cent versus the 48 per cent, Scottish independence and the rest. But there were lots of problems in Richmond Park, and we overcame them. And you know, working together felt good – it felt like the future. The Tories, Ukip and Arron Banks want a different future – a regressive alliance. We have to do better than them. On Thursday, we showed we could.

Could the progressive alliance be the start of the new politics we have all hoped for?

Neal Lawson is the Chair of Compass, the pressure group for the progressive alliance.