An announcement by the Equality and Human Rights Commission (EHRC) that it is launching a formal investigation into antisemitism in the Labour party is one more sign that the controversy cannot be addressed by internal procedures alone. Was it ever solvable through the party’s own efforts? There was a time when I thought it might be.

Even before Jeremy Corbyn was elected leader of the Labour party in September 2015, there was deep disquiet in sections of the British Jewish community about what was perceived as his tolerance for Islamist terrorist groups. Following his election, repeated instances of antisemitic comments in the burgeoning Corbynite grassroots further stoked alarm. The attempted coup against Corbyn’s leadership in June 2016 deepened the problem, with non-Corbynite Jewish party members (and those within the Jewish Labour Movement in particular) becoming the focus of anger from some who supported Corbyn’s transformation of the party.

There has been no shortage of efforts to address this situation. There was the Chakrabarti inquiry in June 2016 and repeated statements by Corbyn and others condemning antisemitism. There have been meetings, both confidential and announced, between Jewish communal leaders and the Labour leadership. There have been rule changes and bureaucratic restructuring intended to improve the party’s disciplinary procedures.

For years I’ve been advocating dialogue as a way to address the crisis generated by antisemitism within Labour. For a long time my working assumption was that hardcore, unrepentant, unredeemable antisemites in the party were a tiny minority, but there was a much bigger group that fell into antisemitic language occasionally or out of ignorance. The first group could not be dialogued out of existence – only expelled – but the larger group might be open to education. What was crucial was to engage those Corbynites who had no history of antisemitism and might be able to exert influence on others. I did have some hope that, through hard work and trust-building, it might be possible to reach some kind of understanding between those who lead the Labour party and Jews concerned about antisemitism.

Not only has nothing worked, but efforts to fix things have themselves deepened the controversy. Meetings between Corbyn and Jewish community leaders have been tense and incomprehending affairs. Institutional investigations and reforms are either seen as a whitewash from the Jewish side (as with the Chakrabarti report) or as an unacceptable compromise with them (as in the 2018 adoption of the International Holocaust Remembrance Alliance definition of antisemitism by the Labour party national executive committee).

Now, with Jewish support for Labour dropping like a stone and accusations that the party is institutionally antisemitic, antisemitism in the party has not gone away and the political dispute over it is worse than ever. There is no reason to think that the EHRC will end the dispute, whatever its findings – things are just too far gone for that.

So what next? There is a way back, but it’s going to take a radical rethinking of what anti-racism means.

We got into this mess in the first place because sections of the left have never been able to reconcile themselves to the fact that the majority of British Jews are Zionists in some shape or form, either self-identifying as such or supporting the principle of Israel as a Jewish state. That fundamental bewilderment, that sense that Jews should know better, has been combined with a love of that significant minority of Jews who are not Zionists. Groups such as the Corbyn-supporting Jewish Voice for Labour, which is largely made up of Jews who reject Zionism, tacitly encourage the sentiment: “Why can’t all Jews be like that?”

Given that the divisions between Jewish Zionists and anti-Zionists are very much out in the open, it is all too easy to pick and choose the Jews one listens to and to damn the rest.

The only way out of this impasse is to recast anti-racist solidarity so that it is completely decoupled from political solidarity

I am not one of those Jews who would argue that members of Jewish Voice for Labour are not really Jews and should be shunned by non-Jews. But there is no way around the fact that, intentionally or unintentionally, they encourage the fantasy that all you need to do to oppose antisemitism is to draw close to those Jews with whom you are in sympathy. This fantasy has exposed under-discussed questions about how anti-racism should express solidarity with minorities who are subjected to racism: what happens when those minorities, or significant sections of them, hold to politics with which you don’t agree? And what happens when those minorities treat those politics as non-negotiable parts of their identity?

Too often, anti-racism on sections of the left is predicated on wilful ignorance about what the victims of racism actually believe. Jews have a way of forcing the issue: our overwhelming (but by no means total) embrace of Zionism has been so public that it cannot be avoided. This has presented a quandary to those who see themselves as supporters of the Palestinians: how can the victims of racism be racists themselves? The way out of that has sometimes been to deny that Jews today constitute a group that can suffer racism at all (other than perhaps at the hand of good old-fashioned Nazis); we have been subsumed into white privilege. The result has been that progressive movements increasingly find it difficult to include Jews who do not renounce Zionism, as the controversy surrounding antisemitism in the Women’s March in the US has shown.

The only way out of this impasse is to recast anti-racist solidarity so that it is completely decoupled from political solidarity. Anti-racism must become unconditional, absolute, and not requiring reciprocity. Anti-racism must be explicitly understood as fighting for the right of minorities to pursue their own political agendas, even if they are abhorrent to you. Anti-racism requires being scrupulous in how one talks or acts around those one might politically despise.

This isn’t just an issue that applies to Jews and antisemitism. We are beginning to see the strains in other forms of anti-racism too, when minorities start becoming politically awkward. The opposition from some British Muslim groups to teaching LGBT issues in school is one example of this. Yet opposition to Islamophobia is as vital as opposition to homophobia and one must not be sacrificed on the altar of the other.

The anti-racism that I suggest is a kind of self-sacrifice. Anti-racists must acknowledge but restrain how they really feel about those who must be defended against racism. Doing so involves a constant balancing act: supporting the right for Zionist Jews to live free from abuse and harassment while, at the same time, fighting for the right of Palestinians to live free from oppression. Creating that balance involves teeth-gritting; choosing not to pursue the most unbridled forms of political warfare when it involves ethno-religious minorities such as Jews.

It sounds like a horrible, frustrating and maddening process. But who said that anti-racism was going to be easy? Well, it isn’t easy and the fantasy that it is got us into this predicament in the first place.

This, then, is what a solution to the Labour party antisemitism crisis will have to look like, now that dialogue and conflict resolution have proved to be dead ends: an acknowledgment from the anti-Zionist left that anti-racist solidarity with those seen as despicable Zionist Jews must be unconditional. This is what I call “sullen solidarity” – and it is the most powerful form of solidarity there is.

Paradoxically, the first step in cultivating this sullen solidarity should be restraining love for those Jews with whom one is most in sympathy. The Labour leadership needs to stop its repeated expressions of support for particular Jewish traditions; its Passover messages about social justice and its invocations of the battle of Cable Street. As a leftwing Israel-critical Jew, I myself honour and respect some of the traditions with which Corbyn empathises, but I don’t need my way of being Jewish to be validated by anyone. Anti-racism should not be a reward for being culturally interesting or politically sympathetic; it should require no justification.

I am totally uninterested in whether the Labour leadership like Jews or what sort of Jews they like. I care only that they will refrain from expressing love for certain kinds of Jews and distrust of others, and that they will defend all of us from antisemitism, however unlikable they might find us.

• Keith Kahn-Harris’s book Strange Hate: Antisemitism, Racism and the Limits of Diversity is published by Repeater Books on 11 June