At 7am on Thursday, polling stations will open in an upmarket corner of west London for the Richmond Park byelection. This odd affair is entirely in keeping with a year of unexpected political events.

They might be the party of government, but the Conservatives are not putting up a candidate. The frontrunner is a self-styled “independent”: Zac Goldsmith, the unfathomably wealthy, roll-up-smoking Tory environmentalist who was until recently Richmond Park’s Conservative MP and the party’s candidate for London mayor.

Goldsmith resigned from his party, and triggered the byelection, in late October in protest against the government’s support for the expansion of nearby Heathrow airport. Strangely, however, the local Tory website carries an email address for anyone who wants to volunteer for his re-election campaign, and he has been joined on his rounds by Tory MPs. Ukip has decided not to put up a candidate, instead lending its support to someone its describes glowingly as “a principled man who was fully committed to helping get Britain out of the European Union”.

Zac Goldsmith resigned from the Commons when the government announced its support for Heathrow expansion. Photograph: Guy Bell/Rex/Shutterstock

Clearly, then, the main parties of the right are united behind Goldsmith. But, at the same time, in Richmond and elsewhere, more and more noise is being made about cross-party campaigns and electoral pacts on the left and centre-left. This might do two key things: first, further the case for changing the voting system; and second, act as a block to the most extreme visions of Brexit and the kind of nasty populism that is not only running riot across the world, but also exerting its pull on parties and politicians supposedly of the centre. It is the second factor that gives Richmond Park an air of drama.

The Liberal Democrats held the seat until 2010. Encouraged by the 19.3% swing away from the Tories they managed at the recent Witney byelection, they are outwardly hopeful that their candidate – an accountant called Sarah Olney, who is also opposed to Heathrow expansion – might pull off an unexpected victory, thanks largely to Goldsmith’s support for Britain leaving the EU (his constituency voted strongly in favour of remaining).

Meanwhile, as part of an informal agreement relating to future elections for local council seats, the local Green party – which got 3,800 votes at last year’s general election – has stood down to improve the Lib Dems’ chances. Their activists talk about Goldsmith’s position on the EU, but also his staunch support for austerity and the nasty, Islamophobic campaign he waged against mayoral opponent Sadiq Khan. The newly formed Women’s Equality party has made the same decision.

Which brings us to Labour. Despite senior MPs and some local members calling for their party to pull out – thus maximising the chances of Goldsmith’s defeat, as I have previously argued in the Guardian – Labour has stuck with the usual protocol. Its candidate is campaigning hard in Richmond Park, leading to fears that he will split the anti-Tory vote. At the local party’s meeting to select the candidate on 4 November, a member called Mike Freedman suggested that proceedings ought to be abandoned. He says he was interrupted by an official sent from the Labour party’s London HQ. “He said: ‘You can’t do that,’” Freedman tells me. “I said: ‘I can.’ He said: ‘Well, I won’t let you. I’ll stop you.’ And he said if we didn’t choose a candidate the party would impose one.”

Clive Lewis, Labour’s shadow business secretary, was one of three MPs who urged the party to stand down in Richmond Park. Photograph: Paul Ellis/AFP/Getty Images

Eventually, after Freedman and half a dozen or so others had walked out, the meeting selected a candidate: the journalist and transport expert Christian Wolmar.

“It just seems potty,” Freedman says. “Which is why I did everything I could to persuade him not to stand and the party not to work for him. But, unfortunately, that’s not going to happen. I’m afraid my prediction is that Goldsmith will sneak it by a couple of thousand.”

This all brings us back to the key issue. The progressive half of British politics remains divided while, in the wake of Brexit, the right seems to have buried its differences, and got on with shaping the future.

Is it finally time for what some people call a progressive alliance?

“How do you stop the authoritarian right – the Tories and Ukip – dominating politics in this country for the foreseeable future? That requires people thinking outside the box. It requires looking at the maths on the ground and saying: ‘How do we stop them?’ The Tories are masters at gaming the electoral system and we’re quite pathetic at it. We constantly stand candidates against each other. They get how the first-past-the-post system works. We think we do, but we don’t act on it.”

These are the words of Clive Lewis, the shadow business secretary, an ally of Jeremy Corbyn and one of the three MPs – along with Lisa Nandy and the shadow City minister, Jonathan Reynolds – who urged the party to stand down in Richmond Park. Theresa May, he reminds me, has a parliamentary majority of 12, and there are 27 Tory MPs who beat Liberal Democrats into second place at the last election, when the Conservative party was still affecting to be socially liberal and pro-European.

Former Lib Dem leader Paddy Ashdown is a member of More United, which intends to back general election candidates who support ‘open and tolerant’ politics. Photograph: Graeme Robertson for the Guardian

“But the Tories have now arguably turned into a hard-Brexit, authoritarian party,” he says. “So, there’s a good chance that an electoral coalition in their constituencies could beat them. But if you have Labour and the Greens and everyone else standing you reduce the chances of that happening. More Lib Dems are better than more Tories. That’s just the reality. You’ve got to be brutally pragmatic.”

The idea of cooperation between parties on the left and centre-left has buzzed around British politics for decades. In 1997, Labour and the Lib Dems came to an informal accord to limit their campaigning in some of each other’s target seats and tacitly encouraged anti-Tory tactical voting (in the Cheshire seat of Tatton, the two parties stood down to make way for the independent Martin Bell, who defeated the corrupt Tory MP Neil Hamilton). Over the past two decades, in Scotland and Wales, Labour has been sporadically in coalition with the Lib Dems and Plaid Cymru. Recent history has undermined the cause: the Lib Dems’ decision to govern in partnership with the Tories and the prospect of Labour coming to an agreement with the SNP at Westminster played very badly with a certain kind of English voter in the election of 2015. Now, though, in the age of Brexit, Donald Trump and a resurgent political right, the idea is back.

Most MPs and party leaders remain either sceptical or hostile. Almost all the supportive noise about progressive alliances comes from grassroots activists and people beyond the main parties – although, as Clive Lewis proves, the idea is slowly beginning to find an outlet at Westminster.

For Lewis, the focus is on delivering a win for the political left in a fragmented political landscape and binding together people from different parties to resist the most drastic visions of Britain leaving the EU, while also opposing nasty rhetoric and policy on immigration. For others, the main issue remains a voting system that gave the Tories unchallenged power on the support of only 24% of the electorate and how cooperation between parties could bring radical change.

This is the basic position of Caroline Lucas, the co-leader of the Green party, who has long made the case for cross-party politics with the help of Compass, a campaigning organisation that grew out of the Labour party and aims to create a more equal, sustainable and democratic society. These days, it also includes people from the Lib Dems, the Greens, Plaid Cymru and the SNP.

‘I don’t think we’re going to make much progress if we simply point fingers’ ... Green party co-leader Caroline Lucas. Photograph: Clive Gee/PA

“This is all very real now,” she says. “My worst fear is that we’ll be consigned not just to five or 10 years of Tory rule, but to 20: yet more cuts, yet more rolling back of the welfare state, [indifference to] climate change and a government that’s immune to the idea of investment in the green economy. And that’s what motivates me to say: ‘How can we look for something better?’”

Her essential idea, she explains, is to try to “hack into this incredibly dysfunctional electoral system and see if we could get more MPs elected who support electoral reform”, with a view to a politics “more in tune with the majority of people in the country”.

For some people, I suggest, the idea of the Greens and Labour cooperating with the Lib Dems might be deeply problematic, given what Nick Clegg and co did between 2010 and 2015. “But I don’t think we’re going to make much progress if we simply point fingers,” she says. “Yes, there are plenty of things the Lib Dems did in coalition with the Tories that I absolutely opposed and wish very strongly that they hadn’t done. But, similarly, there are plenty of things that Labour did under Tony Blair that I wish they hadn’t done. But if we just focus on those things, rather than areas where we can agree, then we leave the way open to a very rightwing form of Conservative rule or something potentially even worse. And that, to me, is incredibly irresponsible.”

She and Lewis think that the push has to come from local people in individual seats, rather than from Westminster. Both like the idea of locally organised primaries, in which candidates from existing parties might stand alongside people with no political baggage and people would select the candidate best placed to defeat the parties of the right. “It’s got to be a bottom-up expression of what people want,” she says. “And I think what’s exciting right now is that there are groups springing up around the country that have no link to any political party where people come together and say: ‘We’ve got to get smarter about how we work.’”

The Lib Dems hope Zac Goldsmith’s support for Brexit, despite strong backing for remain in the constituency, will help its candidate, Sarah Olney. Photograph: Martin Godwin for the Guardian

Lucas cites Sussex Progressives, founded in the wake of the EU referendum and based in Brighton. The group includes activists from Labour, the Lib Dems and the Greens, as well as people who are not members of any party. Since the summer, it has been holding monthly meetings, as well as hosting street stalls and distributing leaflets about the benefits that migrants from the EU bring to the UK. Now, it is increasingly thinking about the next election – which, let’s not forget, might arrive next year – and how to put pressure on parties in seats where the progressive vote is split; Brighton Kemptown, for example, where the Tories beat Labour in 2015 by 690 votes, but the combined vote for the Greens and the Lib Dems totalled more than 4,500.

One of the group’s founders is 29-year-old Green party member Georgia Amson-Bradshaw. “I’d take what I could get,” she says. “I’d prefer that, in any given seat, there was a single progressive candidate. I suppose, if I’m really dreaming big, it’d be really amazing if the Greens, Labour, Lib Dems – and even the SNP and Plaid – could come together nationally around a narrow set of commonly agreed policies.”

She tells me about a similar group in Ely, Cambridgeshire, and about people trying to start up progressive alliance initiatives in Truro in Cornwall, Whistable in Kent, London and elsewhere. “Stuff is definitely happening spontaneously around the country,” she says. “And that makes me feel optimistic.”

These new initiatives are all rooted on the left, but other political startups are focusing their efforts on a swathe of politics that also includes the centre-right. The aim is to resist the worst effects of the Brexit vote, while making a stand for the fundamental liberal, democratic values that the increasingly rancorous political mood threatens.

The first is More United, which takes its name from Jo Cox’s maiden speech, which became a memorial to the murdered Labour MP (“We are far more united than the things that divide us”). It recently launched a crowdfunding drive for at least £100,000, to be spent backing candidates at the next general election who support “open and tolerant” politics. Its public faces include the former Lib Dem leader Paddy Ashdown, the historian Simon Schama, tech entrepreneur Martha Lane Fox – and, waving an unlikely flag for the pop/politics crossover, Luke Pritchard of the slightly faded mid-noughties band the Kooks.

Sussex Progressives is one of several grassroots groups that supports left and centre-left cooperation across the parties.

There is also the Tigger-ish TV presenter Dan Snow, who explains that the aim is to help candidates of what he calls “the progressive centre” – including some Tories – and diminish the chances of those from both the hard right and the hard left (which, he says, will include some Labour believers in “Trotskyite, destructive chaos”). People standing in marginal seats will be invited to sign up to a charter of principles, including support for “a United Kingdom that welcomes immigration, international cooperation and a close relationship with the EU” and “a modern democracy that empowers citizens, rather than politicians”. If more than one candidate in a given seat signs up, More United’s national membership will be balloted on whom to pick. Once all that is complete, the organisation will provide money and local volunteers. “It doesn’t take a huge amount of money to shift the dial locally,” says Snow.

And then there is Common Ground, formed after the EU referendum, which also reckons its target voters will include “many Conservative voters and previous Tory supporters”, as well as people on the left. They had planned an open primary to select the best liberal, pro-European candidate in Richmond Park, but bumped up against the complexities of electoral funding rules, and the fact that, in byelections, organisations other than political parties understood to be backing particular candidates are limited to a total spend of £700. “We’re experimenting – we don’t have a blueprint,” says one of the group’s prime movers, the financial journalist Hugo Dixon. “But we’re hopeful that we’ll find a way of doing the same thing in the future.”

In Richmond Park, Compass has put out a leaflet suggesting that local people have the chance to “vote for new politics”. It doesn’t specifically mention the Lib Dems, but it offers the prospect of people “who want a more equal, sustainable and democratic society coming together to support the progressive candidate best placed to win”. The Greens’ election material makes a similar argument, assuring people that “by not standing in this byelection, the Green party aims to be a catalyst for change, successfully building alliances with other local parties to create a fairer voting system”.

Meanwhile, Labour’s Christian Wolmar is still campaigning. He says the idea of a progressive alliance “only emerged very late”, and that the Labour party was not offered a say in whom any single candidate should be. Moreover, he says he has been reminded during his campaigning that “the notion that all these people who say they’ll vote Labour would automatically vote Lib Dem is really not true”.

That may be the case, I say, but the chances of Goldsmith being beaten might still be maximised if Labour were not in the race. Does he worry that, by standing, he might let in Goldsmith?

Labour’s candidate in Richmond Park, Christian Wolmar, says Labour voters would not automatically vote Lib Dem if the party chose not to run. Photograph: Yui Mok/PA

“I’ve thought hard about this,” he says. “If my votes and the Lib Dems’ votes together add up to more than Zac’s votes, I don’t think that means that Zac would have won if I hadn’t been standing. It really depends on the way the votes go. Of course, I really don’t want Zac to win. But we were not in a position where we could just resign from this. There are vast numbers of people who would have been incensed by that, locally and nationally.”

That said, he acknowledges that he has met Labour supporters in Richmond Park whose priority is trying to keep the Tories at bay. “There are some people who certainly say: ‘My instincts are entirely Labour, but I’m going to vote Lib Dem, because I want to get Zac out,’” he admits. Other Labour people, he says, will not vote Lib Dem in any event. Nonetheless, when it comes to other areas of the country, he believes in “thinking cannily and cleverly”, and perhaps negotiating with the Greens and the Lib Dems.

“But that would still be hellishly difficult,” he insists. “It’s much more complicated than you think. There are a lot of people in the Labour party who would resist it, and Lib Dems who would resist it as well.”

At the last election, the Greens’ candidate in Richmond was a freelance journalist called Andree Frieze. She was going to stand this time, too, but came to the conclusion that the prize is defeating Goldsmith, and all else follows from that.

“It would reduce the Tory majority,” she says. “It would send a message to this government that we’re anti-Heathrow expansion and we’re anti-Brexit. There are four food banks in the borough of Richmond, because of this governments’ policies – they didn’t exist before 2010. Zac Goldsmith might be environmentally green, but he is rightwing every step of the way. He ran a shocking mayoral campaign. He voted for every single welfare cut. He’s a Tory.”

What does she think of Labour’s stance? “It’s appalling. They are living in cloud cuckoo land if they think they can win here. It’s pointless.

“This should be about getting rid of a government that’s destroying this country,” she says. “And to do that … well, you have to get rid of Zac.”