In 2002 I accompanied Tony Benn for a short while on his lecture tour around Britain, watching him speak to packed auditoriums and outselling the magician Paul Daniels in Cornwall by three to one. It was a peculiar time that in many ways presaged the moment in which we find ourselves now.

A year earlier, Labour had just been returned with a huge majority and the steepest slump in postwar voter turnout to date. A BBC survey, Beyond The Soundbite, prompted by concern about that decline in participation, found that, while voters were neither “de-politicised” nor “uninterested”, they were “disillusioned” and “disconnected”. When asked to finish the statement “I would get more involved if … ” more than a third of respondents ticked either “I thought my contributions made a difference” (24%) or “I thought anyone would listen” (12%). And that was before the Iraq war.

So when Benn told crowds, “We’ve lost control of our destiny” and “hope is the fuel of progress and fear is the prison in which you put yourself”, it resonated. He told me he had been receiving a lot of correspondence from people who said they were leaving the Labour party because of Tony Blair’s leadership. “I always give the same response,” said Benn, who remained a party member to the end. “That’s all well and good. What are you going to do now?” Leaving a political party tells the story of what you’ve given up, he said, but what was truly important was what you were moving on to.

Both that trip and those words have kept coming back to me as a growing number of people say that having voted or campaigned for Labour for years, if not decades, they can no longer stomach the party. After a steep rise under Jeremy Corbyn, membership has started to fall from its 2017 peak – polling is volatile, but for both the European elections and in Westminster, Labour’s vote share is edging down.

It’s not difficult to see where that frustration comes from. The fudge on Brexit is most often mentioned, with the party’s ineptitude – or worse, complicity – over antisemitism coming second. What is more difficult to grasp is why, after all Labour has been and done in recent memory, from the Iraq war to the pauperisation of asylum seekers, this would be the tipping point. Labour’s position on Brexit has yet to kill anybody, except through boredom, so why choose this hill to die on?

As a former party member who did not vote Labour for over a decade I empathise with the estrangement. I could not vote Labour nationally when Blair was leader. I made no great claims for that. How could I when it meant I didn’t back the party that would introduce the Macpherson report, the minimum wage or the expansion of nursery provision? But I made no apologies either. I didn’t back the party that introduced tuition fees or presided over a rise in inequality. Then there was the Iraq war – that illegal, ill-planned, ill-fated, ill-judged abomination, that so many demonstrated against and so many were slaughtered in and because of which so many still suffer.

Jeremy Corbyn launches Labour’s European election campaign in Chatham, Kent, on 9 May. Photograph: Daniel Leal-Olivas/AFP/Getty Images

For this refusal to toe the line I would endure tiresome lectures about how my middle-class self-indulgence was snatching food from a poor child’s mouth. About how it was OK for me because I was privileged but some people really needed a Labour government, or that I was throwing away my vote. And then the ultimate threat – the Tories. Metaphorically pinning you against a wall, they would list their shortcomings and ask whether I would rather have Michael Howard or William Hague in charge. It’s the lesser of two evils, they would say, apparently not realising that they were still advocating evil.

I will not repeat those lectures to those who currently feel estranged. They were tiresome then, they are tiresome now. Parties earn loyalties, they cannot command them. Once they start blaming the electorate for not supporting them they are sunk. I understand why people might want to “send a message”, particularly in the European elections, where the stakes seem low. What is curious to me is that those who have waited until now to send a message did not choose to send it before, not least because had they sent a message earlier it might not have come to this.

Party loyalty can be overrated but moral consistency and a sense of proportion cannot

It intrigues me that those who have been in the party for some time have chosen to draw the line here and now. Not because there’s nothing to complain about. Labour’s messaging on Brexit has been awful. First Corbyn says “article 50 has to be invoked now” the morning after the referendum, only to tell Newsnight a week later: “I may not have put that as well as I should have done.” One week, Corbyn claims Brexit can’t be stopped; the next, the shadow Brexit secretary, Keir Starmer, insists it can. When asked how clear each party’s Brexit policy is, Labour comes last with 13%. It deserves that. The policy itself is pretty straightforward. It supports a second referendum if parliament rejects the prime minister’s deal and a general election doesn’t follow. They just can’t get their story straight.

No, it intrigues me because there’s always something to complain about. Is Corbyn’s fudge on Brexit worse than Blair’s certainty about invading Iraq? If someone can vote and leaflet for a party that launched an illegal invasion that led to hundreds of thousands of deaths and instability in the region for years to come because they felt that party had a broader purpose, then by what moral yardstick is the party’s position on Brexit too much to bear? Is it that this particular straw is too heavy or that the camel’s back has just become too frail?

The issue of antisemitism is more delicate. Nobody should feel badgered into staying in a party where they do not feel welcome. And when Jews do not feel welcome in the Labour party because they are Jews then that is a serious problem. This issue has been handled badly and at some point that shifts from a bureaucratic matter to an ethical one of institutional indifference. There are clear moral reasons why anyone, but particularly Jews, might abandon the party.

This mass-sensitisation to and mobilisation against prejudice both within the party and without is to be welcomed. I do, however, wonder where that sensitivity was when senior figures in the party were burqa-baiting, accusing the children of asylum seekers of “swamping” schools, celebrating the Empire and branding the Liberal Democrats as “on the side of failed asylum seekers” while Labour was on “your side” (a byelection campaign run by the deputy Labour leader, Tom Watson). The point here is not to change the subject but to contextualise it. Labour has a history of both fighting bigotry and harbouring and, at times, propagating it, precisely because it is embedded in the society it wishes to transform and contracts the very viruses it aims to eradicate. It is helpful to understand the issue of antisemitism in the party as part of a continuum because it can provide allies rather than set up communal rivalries.

Party loyalty can be overrated but moral consistency and a sense of proportion cannot. If Blair’s modest anthem at his first election was “Things can only get better”, then an honest account of Labour’s moral inventory in such moments is that “Things have definitely been worse”.• Gary Younge is a Guardian columnist