It was when I opened the door to Labour canvassers last year, shortly after Jeremy Corbyn’s three-line whip on the unamended Brexit bill, that I realised I was angry. Here I was, wearing a sweatshirt that read “Hell yes I’m tough enough” (in honour of Ed Miliband) unable to promise my support to the Labour Party. I did not say “I’m French”, but “I’m an EU citizen”, and they knew where this was going: Corbyn had whipped Labour MPs into voting with the government, despite the fact that Harriet Harman’s amendment to protect EU citizens’ rights had been defeated.

About 2.8 million EU citizens living in the UK are of voting age, according to the advocate group the3million. Under EU law, European nationals have the right to vote in local and European elections in their country of residence, anywhere in the EU. The last time the UK’s EU citizens could vote was in mayoral elections on 6 May 2016; a few weeks later, the referendum outcome put an expiry date on this right to vote and left them in legal limbo.

The local elections next May will be the first election since Brexit in which EU citizens will have a say. London’s mayor Sadiq Khan called on Monday for EU citizens to use the vote to “punish” the Tories on Brexit – but many are angry at Labour, too.

“I couldn’t bring myself to support Labour after the referendum”, says Costanza de Toma, 45, who has lived in the UK since 1995. Originally from Italy, she became a dual citizen in 2012 – so she has a vote in all UK elections. “Labour must stop fudging its stance on Brexit and citizens’ rights if they want to win EU citizens’ votes.”

Labour’s lack of a strong message on Brexit has confused even some of its most fervent supporters – sometimes not just voters, but volunteers and campaigners, too. “Labour isn’t trying hard enough. They don’t realise that we have voting potential,” Anjie, a 25-year-old Spanish national working in political consultancy, told the New Statesman (because of her job, she asked we only use her first name). Anjie has volunteered with Labour since 2012 and interned for the party in 2016; but this year, she says, she’s not sure she wants to campaign for them.

Sarah, 23, a Greek civil servant from Edinburgh (who, for work reasons, also asked not to give her surname) is certain she will not. After campaigning for Labour in the 2014 No campaign, then the 2015 general election and the 2016 referendum, she finds the party’s Brexit stance “incredibly frustrating”.

“I was loud and proud with Labour support. It’s heart-breaking.” The most difficult thing, she says, isn’t that the party actively “betrayed” Europeans: “It’s that they don’t even try to do anything.”

EU citizens feel stuck: on one hand, Anjie explains, “the danger of voting for Labour is that our vote can be interpreted as accepting Labour’s current stance” on Brexit; on the other, “not voting Labour could very well help the Tories”. Yet in May, many of them, like De Toma, will “certainly not vote Labour.”

“There’s no way I will vote for Labour again with their current Brexit stance”, says Eve Lo, a Hong Kong national in her 40s who has been in the UK for 15 years with her husband from Greece. Eve has permanent residency and because Hong Kong is part of the Commonwealth she can vote in general elections. She has always supported Labour – until last year.

The Mayor of London’s call to “punish” the Tories has served only to remind some Europeans how little Labour has done for them. “I have huge respect for Sadiq Khan,” says De Toma, who used to be his constituent in Wandsworth, “but don’t think he’s done his due with the leadership.” Although individually, some Labour MPs – and Khan – have been supportive of citizens’ rights, the general feeling among EU citizens is that they have not done enough. “They don’t seem to be listened to by Corbyn”, says De Toma.

Pauline Lennard, a Frenchwoman who has lived in the UK since 1998, agrees: “Chuka Ummuna, David Lammy [and other pro-EU Labour MPs] can speak for their constituencies, but in parliamentary debates, when they’re whipped into voting to support the Tories and their Brexit? As it is, they are lowered voices.” (Lammy defied the whip to vote against triggering Article 50 and Umunna defied the whip over the single market).

There have been attempts at supporting EU citizens within Labour, and they are grateful for that. Anjie says pro-EU groups within the party, such as Labour Against Brexit and Labour Campaign for the Single Market, give her hope. De Toma works with the advocate group the3million, and Labour MEPs, she says, are “fantastic” on citizens’ rights. “We have had support from individual Labour MPs, but there is anger with the official Labour line,” says Maike Bohn, spokesperson for the3million.

Sarah is mainly angry at the leadership’s “halfhearted attempts” at supporting Remain during the referendum campaign. She regrets that Labour never matched its “bold” 2017 domestic pledges with a real pro-EU message. “It’s an open secret that they are fine with leaving the EU”, she says. To Lennard, it’s “pretty unambiguous” that Corbyn is pro-Brexit. Eve Lo thinks so, too: “Corbyn and his supporters secretly wouldn’t mind a hard Brexit to happen and for the situation to turn toxic, just so they can get into power”, she says. “I can no longer relate to the current Labour Party.”

EU citizens will not forget the Labour leader’s personal stance on Brexit – and crucially, they are not ready to forgive it, either. “They seem as uninterested in Europeans as real human beings as the Tories are”, says Niels Koschoreck, a German national who has lived in the UK for 12 years. “They voted against our rights, it’s very hard to re-establish trust”, Bohn, from the3million, explains.

But it’s not just 2.8 million voters who may punish both the UK’s main parties for forgetting them in the Brexit negotiations: they have family and friends, too. “Five to seven million British voters are closely related to EU citizens”, Bohn says. “It’s a really significant number of people.”

The Lib Dems are cited by most as the party that will get the EU citizens vote – punishing both Labour and the Conservatives. De Toma joined the Lib Dems after the referendum, and has found them “much more in tune” with her values. Pauline Lennard used to hesitate between Labour and the Lib Dems, but “since Brexit, it’s a no-brainer.” Koschoreck guesses the Lib Dems will have his vote, too. Eve Lo still wishes Labour would “join forces” with the Lib Dems to reverse Brexit; but as this is “very unlikely” to happen, and she’ll vote Lib Dem. “I have lost faith [in Labour] to be able to achieve anything even if by some luck, they get into power”, she says.

Feeling abandoned by political parties can drive voters away – physically. Costanza de Toma’s family will be moving in the summer. “I have no illusion”, she says. “Even with Labour in power, what guarantees do we have?” Like many Labour-supporting EU citizens in the past 20 months, she wonders: “Surely if you’re a Labour politician, you’d stand for re-instating everyone’s rights, including EU citizens?”

This article was amended on the 23 April to clarify that Jeremy Corbyn did not whip Labour MPs to vote against Herriet Harman's amendment, but instead whipped them to support the goverment's Brexit bill despite the amendment being defeated.