“God, what am I doing in this party?” That’s how Frank Field describes what was going through his head: but this wasn’t when Jeremy Corbyn was leader. This was under his predecessor, Ed Miliband, after Field demanded to know if Labour was responsible for “this huge influx” of immigrants, and his fellow MPs cheered when the leader said “no”. Here was a politician who no longer felt comfortable belonging to his party before Corbyn had even stood for leader, and all because Labour was not sufficiently hostile to immigration.

In his resignation letter, Field cites antisemitism as a factor in his departure. To be clear, antisemitism is a sickening disease, it exists on a fringe of the left and there are some in denial over that fact. Both Labour and the broader left have to do far more: British Jews, tortured by a shared history of 2,000 years of persecution and an attempt to exterminate the entire European Jewish population within living memory, feel genuine anger and hurt. Which is why, in part, Field’s attempt to use an issue of deadly seriousness to wrap himself in the garbs of martyrdom is so transparently cynical.

His resignation has nothing to do with antisemitism. Last month, the government was on the brink of defeat over its nonsensical customs arrangements plans. It may well have collapsed if the vote went the wrong way: Tory rebels were told that a general election could ensue. But Theresa May was saved by Frank Field and three other Labour rebels. That understandably riled Labour members, who pay their subs and surrender evening and weekends knocking on doors because they would quite like to replace a Tory government beholden to Jacob Rees-Mogg with a Labour administration instead.

The local parties of both Field and Kate Hoey near-unanimously voted no confidence in both, and called for the whip to be removed. Indeed, both politicians have a unique talent of being able to unite party activists on left and right: Hoey’s party activists nominated Blairite candidate Liz Kendall in 2015. Field knew he was about to be pushed. Instead, he jumped. It was what he called for Labour MPs to do three years ago if they were deselected, and he wasn’t citing antisemitism as a pretext back then.

Field’s claim that he fled the Labour party in the name of anti-racism is, given his record, certainly audacious. His obsessive anti-immigration agenda once led a Church of England bishop to call him “the new Enoch Powell” – the infamous Tory politician Field has himself praised. Here is a man who has described Margaret Thatcher as a “hero” and was appointed by David Cameron as his “poverty tsar”, swiftly announcing plans to shred child poverty targets.

There are those who angrily decry Corbyn as a beyond-the-pale Brexiteer, now cheering on Field as a principled martyr because he resigned the Labour whip before his local party deselected him for backing May’s extreme Brexit plans. That will do little to shake the suspicion that opposing Brexit is not their main priority: preventing a leftwing government is. There are a handful of other Labour MPs said to be planning on making the same journey as Field.

As it’s announced that a new centrist party has split before it has even been launched, it’s clear any new venture will perish. The best they can hope for is to gift a majority to a Tory party increasingly beholden to Rees-Mogg. What a political legacy to have. In the coming months, the media will venerate those determined to stop a Labour government, and both rightwingers and self-described “centrists” will escalate an increasingly vicious campaign against the left. It is difficult to escape the conclusion that they will do so because they are scared of losing, and understandably so.

• Owen Jones is a Guardian columnist