Labour remainers are in a dilemma. Wherever I go I find lifelong Labour voters – indeed (whisper it) even some elected Labour councillors and probably an occasional MP – agonising over their European election vote in 10 days’ time. In the privacy of the polling booth no one will ever know, with both Momentum-leaners and non-Momentumites united in the same quandary.

This election is all about Europe and our place in the world, nothing else, not a dry run for a general election. Results will have only one meaning: how many voted remain and how many voted leave? This will tell if all the opinion polls are right. Not one has shown a leave lead for more than two years, with remain solidly ahead. Will that be replicated on 23 May? The number of seats won by any party will barely matter, compared with the crucial total of in/out votes. This is our second referendum by proxy.

Look at the astounding YouGov poll today and take in the first shocks – the Brexit party on 34%, Labour next closest at a meagre 16%, the Tories flattened to fifth with a near-death 10%, and a surge for Liberal Democrats and Greens. But never mind individual party tallies: all that matters is the number of pro-remain, anti-Brexit votes. And that hangs on which side Labour’s vote is counted.

When the actual votes are in, as things stand now the only ones counting as unequivocal remainers will be those cast for the “Bollocks to Brexit” Lib Dems, the Greens, Change UK, SNP and Plaid Cymru. As of now, Labour votes will not count in that tally – or worse, according to Gina Miller’s Remain United, Labour votes will be counted in the leave column. Voting Labour will not be a vote for remain – as the Brexiters will surely, and rightly, trumpet when results come in.

Keir Starmer says that as many as 150 Labour MPs will never vote for any deal without a public vote, with most Labour MPs on the remain side. But it’s no good claiming, as Tom Watson did valiantly today, that a Labour vote would be a vote for “remain and reform”. That’s his sincere wish, but not, alas, a fact. Absurdly, on Sunday Labour sent the infuriating Barry Gardiner on to Sky to declare the party is for leave. When Sophy Ridge asked, “The Labour position is that we leave the EU?”, he replied, “That’s what we’ve been trying to do with these negotiations with the government. That’s why we’re there.” Labour just wants an “alternative way” because “people are angry that the referendum result has not been delivered by an incompetent government”. He ignores the many more people distraught by Brexit.

Simultaneously, Labour’s estimable Jon Ashworth was trying vainly to hold the fort for remain on the Andrew Marr Show by reminding us that Labour has voted three times in the Commons for a confirmatory vote. But what can Labour people do when its fence-sitting leader proclaims: “We are not trying to win the votes of just leavers, or just remainers. Instead we’re reaching out to everyone. Labour will never be the party of the 52% or the 48%.” What a useless voting guide that is for more than two-thirds of Labour voters and almost all Labour members who want to vote remain. All the signs are that Labour risks haemorrhaging votes to Greens, Lib Dems and maybe Change UK. Labour MP Ben Bradshaw says 12% of Labour’s 2017 vote is defecting to Brexit parties, 48% to remain parties.

But all is not yet lost. There is still time for Labour to pivot towards wholehearted support for a referendum, backing remain and reform. The pointless Brexit talks with the government will break down soon, though neither side wants the blame: the public always wants politicians to sit down and agree, even from irreconcilably incompatible positions. When talks end, that’s the time for Labour to declare with absolute honesty that they did what they promised their Brexit voters: they tried to get a permanent customs union and single market alignment. They hoped for a general election but plainly there’s no time, so they now obey their commitment to take the question back to the people. There is no one else to resolve this but voters themselves. Labour can trounce the absurd claim this would be an “undemocratic betrayal” when it’s the Brexiters who dare not ask the people, fearing the dusty answer they would get next time.

There is no knowing if Jeremy Corbyn is capable of this agility or too stubborn to shift. But when driving towards a concrete wall, a swerve is advisable. Some obstacles have been removed. The People’s Vote campaign was viewed by some Labour conspiracy theorists as a way of covertly rallying anti-Corbynites. But now the Change UK group has split off, so far making very moderate headway, the path is clear for Labour remainers to be seen as just that – pro-Labour, pro-referendum, pro-remain, no subversive agenda. In this election Labour’s remainers will put country before party – and vote accordingly. The party’s Brexit voters have already fled. It’s for Corbyn to decide if he wants Labour humiliated: his remainers are waiting to hear, before they too defect.

The threat of Nigel Farage – a bully, someone who consorts with worldwide far rightists, racists, antisemites, brutes of every variety – is real and frightening. Sending his rogues’ gallery of the comic and the extreme to Brussels will be an embarrassment – but the headline that matters will be total votes cast for staying in Europe. If Labour is not solidly on the side of the remainers, to its shame its voters will be scooped up and counted among the Brexiters. That mortal error could ensure first-time defectors never return.

• Polly Toynbee is a Guardian columnist