Theresa May’s speech to the Conservative party conference last week will go down as one of the most calamitous in modern political history.

Nevertheless, the accolade for the worst speech she has ever delivered goes to her conference address almost exactly a year earlier, when she declared: “If you believe you’re a citizen of the world, you’re a citizen of nowhere. You don’t understand what the very word ‘citizenship’ means.”

Many people like me, who are either the descendants of immigrants or first-generation immigrants themselves, found the speech chilling. A lot of us have complex layers of identity with different parts coming to fore at the different moments. We devise labyrinthine sets of rules governing sporting loyalties, with support for one side or the other dependent on a complex analysis of the circumstances. None of this is incompatible with being proud, loyal, hard-working and law-abiding citizens of Britain. But May dismissed all of this in a single, brutal phrase. For her, identity is completely binary and a refusal to come down on one side or the other means that you are nobody.

May faced the consequences of her speech on the night of June 8 2017. If BAME voters had behaved as they had done in the 2015 election, then the Tories might have won 28 seats that they actually ended up losing. In other words, if she had kept ethnic minorities on board then the Tories would currently be sitting on a 42 seat majority.

This week the government’s race equality audit was published. As London Young BAME officer Abdi Duale argued on Tuesday, May deserves praise for commissioning the report, but there are serious doubts about whether the Tories will do much to address the issues raised in it. The prime minister has a track record of talking about progressive-sounding ideas – see also industrial strategy and workers on company boards – then doing nothing about them. The audit in itself is unlikely to change the negative attitudes of BAME voters towards the current government.

It all seems a long time ago since the 2015 election, when the loss of BAME votes to David Cameron’s Tories was flagged up as a significant cause for concern in Labour circles.

In the past, minority ethnic voters had overwhelmingly plumped for Labour as the party which had consistently campaigned against racism and which in government had legislated to combat discrimination in all forms, usually against Tory opposition.

However, by 2015, there was no reason to assume that BAME voters felt that they owed such an historical debt of gratitude to Labour that they were bound to give us their votes in perpetuity. Whilst Cameron and George Osborne made their contempt for the less well-off very clear, their contempt was at least wide-ranging and therefore reasonably colour-blind. In 2015 wealthier BAME people in particular started to vote Tory. But with Cameron gone, and his “modernising” project disappearing along with him, this June those BAME votes came straight back to Labour. After all, nobody backs a party which seems to fundamentally dislike who they are.

But Labour must also acknowledge that this is not just a problem for the Tories. Between 2015 and 2017 the percentage of working class people who voted Tory jumped from 27 per cent to 45 per cent – and the 2015 election came after decades in which Labour had already seen a significant erosion of its working class vote. In the same way that the Tories have alienated many BAME voters, we must accept that Labour has alienated large sections of the white working class. Meanwhile, the race equality audit not only drew attention to problems affecting BAME people, but also highlighted the gap in attainment between poor white children, and poor children from other ethnicities. It is not just minority ethnic groups which are not achieving their full potential in modern Britain.

The fact is, that any party which comes across as intrinsically hostile to a whole demographic group is going to struggle electorally. But in the divisive political space we currently occupy, there is an assumption that helping one set of people must be taking something away from someone else. This is not true. It is to the detriment of the entire country that so many people from ethnic minorities are unemployed, just as it is to the detriment of the entire country that poor white children are underperforming at school. Britain can only be made a better and fairer country by dealing with all of these issues.

The Tory problem with BAME voters will clearly not be solved by the publication of the race equality audit. But if the Conservatives stop squabbling with themselves long enough to elect a leader who does not go out of her way to insult minority ethnic voters, and who appears sincere about tackling the problems highlighted in the audit, then Labour equally needs to make sure that we reach out to demographics we currently struggle with. Otherwise BAME voters might well end up handing the Tories’ majority back to them.

Christabel Cooper is a Labour activist in west London and a council candidate for May next year.