Norman Tebbit probably did not realise it at the time but an idea he floated in a Los Angeles Times interview in April 1990 is still hurting the Conservatives today. Internal party focus groups have highlighted the "Tebbit test" (when it comes to cricket, who do you cheer for?) as a problem among Hindu and Sikh men in their 50s and 60s.

The party discovered that his suggestion continue to deter these voters, many of whom have instinctively conservative values, from voting Tory. These British men don't want to be labelled as foreign for shouting for India this weekend.

Conservative history is littered with Tebbit-like moments and much worse. Just think Enoch Powell, the "If you want a nigger for your neighbour, vote Labour" slogan used in Smethwick in the 1964 general election or apartheid. And that while the Labour party was opposing segregation in South Africa and linking itself to landmark anti-discrimination policy.

The combination has created an electoral headache for David Cameron. When Baroness Warsi, the former party chair, stuck the knife in after resigning as a minister over Gaza, she was right in one sense. The Tories do have a deep-seated problem when it comes to the "ethnic vote".

Party strategists wince at the results of the ethnic minority British election study, which analysed the 2010 figures. Just 16% of black and minority ethnic groups voted Conservative, compared to 68% backing Labour. To put that in perspective, it means Mr Cameron did worse among minority groups in Britain in 2010 than Mitt Romney did among non-whites in 2012. And that was a US election failure that has left the Republicans wondering if their party can survive a fast-changing population.

Mr Cameron will be relieved to know that the British non-white population is 14%, significantly lower than in America. But when you consider the similar demographic trend – one in four under-10s is non-white – you start to grasp the scale of the Tory challenge. One senior adviser fears that securing 16% of this growing vote in 2020, 2025 or beyond will not just squeeze majorities but crush the party. The party faces an "existential threat" if it fails to increase its appeal dramatically, he tells me. "It has taken years to get us into this mess. It will take a long time to unpick."

The frustration for the Tories is that they might expect some of these groups to choose them over Labour. The election study found that while better-off white people were significantly more likely to vote Conservative than their less wealthy counterparts, the same was not true for non-whites. That is despite the fact that minority groups were more right wing than the majority on the key issue of tax and spend. The one silver lining is that minorities, obviously, do not vote as a monolithic block; while only 6% of African background voted Tory, 24% of Indian background did.

Still, the picture facing Conservatives is that high-income people, politically on the right, who want a smaller government (and a tough stance on crime and immigration according to other studies) are still much less likely to vote Tory if they are non-white. Whatever the offer, they simply think this is not a party for people like them. Many of these groups have values that Conservatives consider closely aligned to their own. The view put forward by former prime minister John Major last week, about immigrants having a "very Conservative instinct", is shared at high levels. One senior figure thinks the aspirational values of anyone who uproots their family to move to Britain should chime with the party.

Now the party is trying to mirror an aggressive ethnic-outreach effort carried out in Canada. Several meetings have already taken place with politician Jason Kenney, who spearheaded an effort that drove the support of "visible minorities" for his Conservative party to 31%. "Just imagine what it would mean if we could do that here," said one politician, arguing it could also cut into Labour's base. Sources say the lessons they have drawn from Canada are threefold: first, that "showing up" at community events matters; second, that it is imperative to engage heavily with specialist minority media; and third, to mind your language.

"We were absent; now we are everywhere," says one source on the first point. They comment on how the Curry awards were always attended by a plethora of shadow cabinet members. This year, Mr Cameron spoke. He also travelled to the holy Sikh Golden Temple in Amritsar and took his wife, Samantha, dressed in a traditional sari, to a Hindu Diwali celebration in northwest London. The same is being asked of MPs and candidates. Not just a photocall but ongoing, serious engagement.

Advisers insist it is more than window-dressing, arguing that turning up shows respect but also provides the opportunity to listen to and act on what Mr Kenney calls micro-issues. One example picked up by the Tories is health and safety rules linked to turbans in the workplace. On a wider scale, policies on forced marriage and stop and search are part of this drive. Meanwhile, the party has hired full-time staff to engage with specialist ethnic minority media. And that is not just print media such as the Eastern Eye and the Voice, but Asian television channels from which many people get their news. Tory staff scrutinise the number of stories breaking through in specialist press and broadcasting, compared with Labour, and send that weekly information to all MPs. They believe they are doing well.

As for language, I remember feeling uncomfortable when the prime minister declared in a speech that multiculturalism had failed. He was using the "Whitehall definition" of multiculturalism, which he described as a policy that concentrated on the differences between ethnic communities. Yet many people didn't see the nuance and felt he was attacking the very nature of diversity. Polls suggest it went down badly with ethnic minorities. Another interesting point is that even when people support an idea (many minorities take a tough stance on immigration for example), finding out it is a Tory policy puts them off. The polling that revealed this tendency in 2005 was a central part of the Conservative modernisers' arguments to try to detoxify the brand.

This ethnic engagement is a longer-term part of that drive, one that some think could take decades to complete. And yet Labour would argue that any such plan has already been railroaded by the threat of Ukip and the rising influence of rightwing backbenchers. "This is not just about Enoch Powell and Norman Tebbit. Its is also about modern-day Tories like Lynton Crosby, Gove and Cameron himself," said a senior source.

Labour insiders think voters will have been put off by modern-day Tebbit tests. They talk of Home Office "go home" vans, the characterisation of the Trojan horse investigation in Birmingham schools and the Tories aligning with far-right parties in Europe. "We have spent 30 or 40 years earning trust, fighting racism and standing up for communities," says the senior source. "The history of the Conservative party is not a pretty one. You can't undo that by attending a few events." Still they know they can't be complacent. Even small successes by the Tories could hurt.

Others within the Labour party, and also some Tories, criticise the role of Lynton Crosby, the party's chief strategist, charged with winning the 2015 election. They feel that failure to look beyond next May means he is happy to let the ethnic vote drop. It is true that many Tories felt that the work needed to re-engage particularly distant ethnic groups, such as black Africans and Pakistanis, was not something that would be achieved before next May. In a small way, that did feed into the decisions on which seats should be part of the so-called 40-40 campaign – those that central funds would be poured into in a bid to hold or gain them.

Although most of the focus was on factors such as majorities, incumbency, the Labour candidate and so on, consideration was given to the ethnic make-up. If a majority was only possible with a significant boost in votes among the most difficult-to-reach groups, the seats were not chosen as targets this time. Yet Mr Crosby does appear to support this drive to reach out to new, non-white potential Conservative voters. For example, in those 80 target seats, election agents have been dropped to make way for campaign managers recruited, paid for and trained centrally. At a residential course in Solihull, the men and women leading the most marginal fights are being shown the figures relating to ethnic minorities and told how to act on them urgently. Advisers have got more pressing concerns than the future. According to the thinktank British Future, it is "the ethnic gap" that means Mr Cameron is governing without a majority now.

Anushka Asthana is Sky News political correspondent