For ethnic minorities, voting in Britain has always been as easy as ABC: Anyone But the Conservatives. Even in years of Tory advance such as 1983 and 1987, the party has never outpolled Labour among any non-white group, regardless of age or income.

During the era of Thatcherite hegemony, that fact didn’t matter all that much. Ethnic minorities made up a very small proportion of the electorate and they were more concentrated in certain parts of the country, which made them less electorally influential. But there are now three million ethnic-­minority voters and they are increasingly dispersed around the country.

Labour’s historical dominance over the ethnic-minority vote was such that even in areas of increasing wealth, it remained competitive and in some places it was hegemonic. After the 2005 election the more optimistic members of the leadership dreamed of a future in which Britain’s changing ­demographics made Conservative victories not just rare, but impossible.

As Tory leader, David Cameron made it his personal project to defy that trajectory. To assist in the process, high-flying ethnic-minority candidates had their paths to selection eased – even those, like Kwasi Kwarteng, who had little time for Cameron’s flavour of Conservatism. The aim was to give a sense that the new-model Conservatives were a party for all Britons.

On the campaign trail in both 2010 and 2015, Cameron visited Hindu temples and Sikh gurdwaras, and became the first sitting Conservative prime minister to be interviewed by the Voice, Britain’s only national black newspaper. “I think lots of black Britons look at the values of the Conservative Party and think we agree with you about family and community . . .” he told its reporter Biz Pears. “The holdback has been people asking themselves if they can get up and get on with the Conservative Party, and you can see now that you can.” His home secretary, Theresa May, attacked the police for its overweening use of stop-and-search.

Although these front-of-house changes transformed the Conservative parliamentary party, they did less to transform the Tory vote. In 2010 the Conservatives secured 36.1 per cent of the vote across the country but underperformed that figure among ethnic minorities all the way up the income scale, contributing to the hung parliament. Even in 2015, the few disappointments for the triumphant Tories came in places where ethnic minorities were clustered: Ealing Central and Acton, Ilford North and Wolverhampton South-West. (As for Labour, the party became noticeably more reliant on ethnic-minority votes as some of its white voters moved to Ukip.) Cameron’s new approach did go some way towards lessening the Conservative Party’s ethnic penalty: the increased Tory majorities in Northampton North, Harrow East and Gloucester were won with the votes of affluent ethnic minorities.

A Survation poll for the think tank British Future put a figure on Cameron’s success: a third of all ethnic minorities voted Conservative in 2015, adding a million votes for the Tories. Most promisingly for them, for the first time among mixed-race voters, Labour failed to secure an outright majority, its share falling to 49 per cent (with the Conservatives on 26 per cent).

Other countries have more racially polarised splits in voting behaviour. In the United States, black voters backed Hillary Clinton by 88 per cent to Donald Trump’s 8 per cent: in the UK the Conservative share did not drop below 21 per cent among any minority group.

But Cameron’s greatest achievement among minority voters was yet to come: a year later, a landslide majority of ethnic-minority people turned out in an effort to maintain Britain’s membership of the EU. While white voters backed a Leave vote by 53 to 47 per cent, two-thirds of all Asian voters opted to remain, as did close to three-quarters of black voters and seven in every ten Muslim voters.

The votes of the brown were not enough to save Cameron from the preferences of the white. Which may indicate that, even if the polls showing a Conservative landslide in 2020 come to fruition, his electoral performance among minority voters will remain the high-water mark for a Tory leader.

As successive inquests into the Conservative defeats of 1997, 2001 and 2005 all found, the British right’s minority problem was decades in the making: not just Enoch Powell, but Norman Tebbit’s famous question about whether or not second- or third-generation immigrants truly supported the England cricket team. Even minority voters who responded to promises of lower taxes feared that a Tory government would never seriously seek to increase prosperity among minorities, however affluent they were.

In Theresa May, the Conservatives have an intriguing centre-forward. She was the home secretary who challenged the police on stop-and-search and who, as Prime Minister, has spoken explicitly about the colour bar to the highest professions. (“If you’re black, you’re treated more harshly by the criminal justice system than if you’re white,” she said outside Downing Street on 13 July 2016.) But she is also the former home secretary who sent out vans calling on illegal immigrants to go home and the Prime Minister who said “if you’re a citizen of the world, you’re a citizen of nowhere”.

It is true that most second-generation and third-generation immigrants share the scepticism towards immigration of many of their white peers. But what David Cameron understood is that when Conservative politicians talked tough on border control, ethnic-minority voters often heard the harsh tones of the far right. He understood, too, that many ethnic minorities in this country feel they have a double citizenship, whether legally or culturally.

Given the likely shape of British politics in 2017, Theresa May will find it difficult to sustain her party’s appeal to ethnic minorities while appeasing the Brexiteers’ desire for tough measures on immigration. If she fails, British (and especially English) politics could begin to polarise more sharply along racial lines.