As Labour foreign secretary in 1951, Herbert Morrison described independence for African colonies as comparable to giving a child of 10 “a latchkey, a bank account and a shotgun”. In 1955, Winston Churchill told his cabinet he wanted to fight the general election on the slogan: “Keep England White.” In 1964, in Smethwick, West Midlands, a Tory candidate bucked the national trend to win with a campaign slogan: “If you want a nigger for a neighbour vote Labour.” In 1968, Enoch Powell warned there would “rivers of blood” if non-white immigration were not halted. In 1978, Margaret Thatcher referred to Britain being “swamped by people with a different culture” – a phrase repeated by David Blunkett in 2002 in reference to asylum seekers’ children in schools. In 1990 three Liberal Democrats in Tower Hamlets were expelled from the party after distributing a fake Labour leaflet alleging Labour would rehouse homeless Bangladeshis. In 2004, the agent in a byelection campaign, chaired by Tom Watson, emailed the candidate, Phil Woolas, and his campaign manager: “If we don’t get the white vote angry he’s gone.” When a court ruled that Woolas had breached election law by making false statements he received messages of support from both Gordon Brown and Cherie Blair. In 2009, the British National party won two seats in the European parliament. In 2014, Nigel Farage, then the leader of Ukip, said “the basic principle” of Powell’s rivers of blood speech was “correct”: his party topped the poll in the European elections later that year.

The current prime minister has refused to apologise for referring to black people as “piccaninnies” with “watermelon smiles” and Muslim women as letterboxes. A single black MP, Diane Abbott, receives half of all the online abuse sent to female MPs.

Racism is not new to British politics. How could it be? As a nation built, in no small part, on slavery and colonialism, it has long infected our culture and institutions. And since politics emerges from that culture, is embedded in those institutions and involves distributing resources and power, it could never be immune. It is not just a matter of individuals with bad ideas, but a system of oppression that does bad things. All parties navigate it – you can’t be on the ballot and not be in society.

For those who imagined that electoral politics offered a clear moral choice between parties untainted by racism, the last week would come as a particular shock. It certainly seemed to throw ITV’s political editor, Robert Peston, in his response to the chief rabbi’s statement that Jeremy Corbyn’s handling of antisemitism made him unfit for office. “I am not making any kind of political statement here,” he tweeted. “What I am saying is our democracy has traditionally been a beacon of tolerance and understanding.” Peston argues not only from a position of absolute ignorance but relative privilege.

It’s difficult to see what meaningful conversation you can have about racism in British politics that does not involve the Windrush scandal

This is, of course, a very political statement from him. The fact that a political editor believes it is not is itself revealing. It seeks to take racism out of the realm of politics and into the domain of decency and fair play. Looking beyond politics to the coarse discourse below, he sets himself above it all and bigotry beneath us all.

Racism doesn’t work like that. It is not a metaphor – a symbol for incompetence, boorishness, impoliteness or stupidity. Nor is it a state of mind – a deficiency of the soul, heart, intellect, emotion or spirit. Racism is a systemic form of discrimination, with a centuries-old legacy, that shapes lived experience today. As such, it cannot be weaponised because it is already a weapon. As well as denying people employment, housing, education, equality, human rights, safety and opportunity, it can literally kill.

Racism can, however, be deployed. It may galvanise, distract, deflect, distort, scapegoat and marginalise. It is an incredibly effective tool for dividing people and giving a sense of superiority to those to whom you have nothing material to offer.

Jeremy Corbyn and Boris Johnson: ‘When it comes to combating prejudice, people expect more from Labour; they expect nothing from Conservatives.’ Photograph: WPA Pool/Getty Images

The fact that it has blighted both parties does not mean they are equally as bad. Labour has, historically, been most likely to identify with movements that opposed racism – even if it has not always lived up to its promise – and to pass laws that supported equality. That is what makes the moment so distressing. When it comes to combating prejudice, people expect more from Labour; they expect nothing from Conservatives.

Nor does it mean that nothing can be done about it. The racism that exists in politics can only be eradicated through politics. At any given moment there are potential allies and opportunities to force institutions to meet their historic and stated ends. Trade unions were once the principal obstacle to non-white people being hired in some sectors in this country. Minorities did not abandon unions. They faced them down, fought them from the inside and turned many of them into effective bulwarks against racism. The issue is whether the organisation in question is fit for purpose.

The aim here is not to change the subject from antisemitism and Labour, but to broaden our understanding of it and, in so doing, deepen our capacity to resist it. It is, of course, possible to discuss antisemitism in Labour on its own terms. It is not necessary to deflect attention from it, minimise it, deny it or debate it. The presence of an element of the left that is susceptible to antisemitic conspiracy theories, and anxiety among Jews that those views are being tolerated and even encouraged, are both real.

It should also be possible to discuss it in relation to antisemitism elsewhere and to racism of other forms. Indeed, it is important to do so lest it is rendered an abstraction – examined as an isolated and particularised case of prejudice in the polity that afflicts one party and one minority, rather than a longstanding challenge to a part of our politics and culture that affects many minorities and all parties.

To talk about Islamophobia in this context is not “whataboutery”. Half of all religious hate crimes in England and Wales are directed against Muslims, the religious group most disproportionately underrepresented in parliament and overrepresented among the poor. The Tory party’s record on Muslims is appalling and it has no intention of addressing it specifically. It’s difficult to see what meaningful conversation you can have about racism in British politics that does not involve the Windrush scandal, which saw British citizens forced out of jobs, denied housing and health care, deported, or threatened with deportation by a knowingly callous government. The only way in which issues like this become irrelevant is if you seriously think that a “beacon of tolerance and understanding” has only now been extinguished.

It’s an election. We have to make a choice. If “the soul of our nation” really is at stake then these too are factors in that choice.

It is possible to keep more than one idea in your head at the same time. Indeed, it is necessary because those who would seek to divide us would love us to remain in ethnic silos, arguing about who has it worse – rather than uniting to make it better for everyone.

• Gary Younge is a Guardian columnist