Racism isn’t much understood in Britain in 2020. Exhibit A is Trevor Phillips suggesting that because “Muslims are not a race”, there is no such thing as anti-Muslim racism.

Phillips has been suspended from the Labour party for alleged Islamophobia, but his line of thinking is common. Many have tried over the decades to explain why it isn’t right, and not just when it comes to Islamophobia. Of course Muslims are not a “race”, but they have been racialised. This means people who are Muslim are treated by particular societies as having essential characteristics that mark them as wholly different.

The question is whether there is a set of attitudes and behaviours – racism – that is socially widespread and used to justify discrimination against a particular group. If we look at the history of racism itself, we see that it didn’t emerge through scientific investigations or empirically rigorous endeavours, but instead to provide a justification for the economic, political and social domination and exclusion of particular groups. This is why it makes sense to call antisemitism and Islamophobia forms of racism.

Last week, the Muslim Council of Britain (MCB) submitted a document to the Equality and Human Rights Commission citing further incidents of Islamophobia within the Conservative party, noting that it was “regrettable” that no action had been taken to solve a “systemic” problem. The extensive review shows why the question of racism in political parties isn’t only a matter of a few bad apples, and that how the bad apples become rotten isn’t simply due to casual or poorly formulated, bigoted views.

This takes us to exhibit B. Perhaps a bigger barrier for understanding and responding to racism, including Islamophobia, is treating it as a fringe or marginal issue only perpetrated by far-right individuals. If we talk only about individuals, we end up with a conversation where respectable people with public platforms or large political majorities couldn’t possibly be personally racist and where it’s offensive even to suggest otherwise.

So far the Conservative party response, when not denying the existence of the problem, has been to suggest the focus should be on “anti-Muslim hatred”, a deliberate attempt to suggest that the question is about bigoted, irrational or violent views. This framing is also seen in the government’s policy response, where the focus is on the far right and hatred, or personal prejudice, which inevitably places the focus on particular individuals, but not on policies or institutions or wider social issues.

In fact, racist views are always based in widespread, relatively sophisticated tropes about particular groups. For Muslims, we all know what these are without even reading the MCB dossier: that Muslims are inherently different, hold illiberal views, and are likely to commit violence, and are a civilizational (or “demographic”) threat going back centuries. There’s nothing new or creative here; with racism it’s always the same old tropes.

Islamophobia in the Conservative party (like racism in all political parties) is based on wider social attitudes – views that end up justifying discrimination against all Muslims. In this sense, racial or ethnic stereotypes are structural or societal in nature, and people may legitimately be unaware where they picked up such stereotypes and may genuinely not intend to say or do racist things.

This is why it’s important to focus on the wider social or structural causes of racism, and to avoid making the conversation about racism about whether a particular individual is a racist. Take Rory Stewart’s comments about young black people or a widely shared mural of Jewish people. Even if you hadn’t seen or heard about these particular cases, you can probably guess what sorts of stereotypes about black men or Jews were deployed. This suggests the question isn’t whether a politician or media commentator intended to be racist, but how and why they (and all of us) are so likely to pick up the same stereotypes, and how and why those stereotypes may continue to be used to justify racial inequalities in 2020. For example, about 44% of the UK population believes that some ethnic groups were “born harder working”.

When it comes to understanding racism, there’s another wider problem: class. The focus on far-right street violence tends to go along with the idea of working-class communities as the last residue of racism in Britain. This is not just misleading, but self-serving denialism from middle-class Britain. This patronising, divisive stereotype about working-class communities ignores the fact that they typically have more social mixing across ethnicity.

It’s understandable that many British institutions dominated by the middle class would prefer to talk about “anti-Muslim hatred” and pretend that’s all there is to say about racism. But it’s not working-class people who make ethnic minorities send out 80% more CVs to get an interview, who design Treasury budgets that hit the poorest black women hardest, or who award fewer firsts to ethnic minority graduates than white graduates. Instead it’s middle-class graduates in decision-making positions, who have done relatively little to address these inequalities.

Racism in political parties doesn’t arise out of a void. The views and attitudes expressed by members of political parties instead flow from deep roots in our society. Those roots are being watered every day in Britain, with the MCB report showing a systemic network of anti-Muslim racism that is more than the sum of the individual cases. Until and unless racism is understood in systemic terms, including in Britain’s party of government, we’ll continue to misunderstand what racism is, and how we might better tackle it.

• Omar Khan is director of the race equality thinktank the Runnymede Trust