Frank Field has criticised local Labour party members after he became the latest party MP to lose a confidence vote for siding with the government in key Brexit votes.

The veteran Eurosceptic’s constituency passed a vote of no confidence against him on Friday, after a similar motion against fellow Brexiter Kate Hoey in her Vauxhall seat in London.

Field has hit back, accusing local members of trying to misrepresent his votes in support of the government’s Brexit policies to try to get rid of him.

The no-confidence vote does not carry any official force, but constituency party members could seek a trigger ballot that would have the potential to deselect Field.

Field said his decision to support Theresa May’s plans in defeating an amendment calling for a post-Brexit customs union had been on behalf of “millions of Labour voters, mainly in parts of the country that have long been neglected by the elites, who gave politicians a clear instruction to take the country out of the EU”.

Field is one of four Labour MPs, including John Mann, Graham Stringer and Hoey, who backed the prime minister’s Brexit agenda. Their critics argue that had the Tories lost the vote, which it won by a margin of three, May might have been forced to call a general election.

Field’s constituency in Birkenhead, Wirral, voted in favour of leave in the 2016 EU referendum.

He said that during his 39 years as an MP he had “always voted to free our country from the tightening stranglehold of the EU” on behalf of working-class Labour voters and that it was important to do so now.

“For most, if not all, of those votes I did so alongside Jeremy Corbyn and John McDonnell,” he said “It would have been a betrayal of the principles I have held for my entire political life, had I voted against the legislation two weeks ago.”

Both Labour and the Tories committed themselves to honouring the leave vote during the 2017 election, he said, meaning “such legislation would have been introduced by whichever party was in power”.

Laura Parker, the national coordinator of Momentum, had called for the deselection of all four rebels, saying: “There is no room for Labour MPs who side with the reactionary Tory establishment.”

Like Hoey, Field faces the risk of being deselected after losing the confidence of local members. The vote against him was sparked by a letter from the Bidston and St James branch secretary, Brian Parsons.

Parsons attacked the “actions and total disregard” of Field in supporting the Conservatives, which he said could potentially damage support for Labour in any upcoming election.

“For whatever reason, Mr Field is refusing to assist his parliamentary colleagues in removing one of the cruellest and most savage Tory administrations this country has produced, including the Thatcher government,” he said.

Field dismissed the idea that he was backing the Tories or “getting in the way of an early general election”. “This is an absurd suggestion that is being used as an excuse by certain people in the Birkenhead constituency Labour party who are fixated on the idea of trying to get rid of me. They would have found some other excuse, had it not been this topic, as they’re already lining up their alternative candidate,”he said.