Nigel Farage has had another bad week. A series of new revelations have appeared in the media focused on claims of sexual harassment at the top of Ukip and homophobic and offensive remarks by a Ukip candidate in Essex, who has since resigned. There is no doubt that Farage will be feeling frustrated. These are the kinds of problems that a host of internal changes implemented over the summer were designed to remedy. Now, many are again pointing to the revelations as evidence why Ukip cannot last as a significant political force. But are they right?

We have been here before. Earlier this year, and shortly before the European parliament elections, there was an explosion of media coverage that focused on the unsavoury (and, in some cases, openly racist or homophobic) views of certain Ukip supporters. Numerous articles across the national media drew attention to the remarks of Ukip candidates on everything from Mo Farah to rape and gay people. “Ukip,” said David Cameron, “have condemned themselves during this campaign with a succession of pretty unpleasant remarks. The whole country has heard enough to know what sort of party it is.”

Less than 10 days later, Ukip won the European parliament elections. Farage’s working-class revolt against the EU, Westminster and immigration attracted more than four million voters and almost 27% of the national vote. Those who supported Ukip either had not seen the coverage or simply did not care.

Following the election we were told that this was just an angry protest vote, that Ukip would quickly implode as the minds of voters turned to the more serious business of who should represent them in Westminster. But still Ukip’s revolt trundled on. Support for Ukip remained stubbornly persistent in the opinion polls at more than 15%, which is around twice the level it needs to inflict serious damage on the main parties next May. Farage went on to coordinate two successful parliamentary byelections for his party, both of which saw the main parties and media devote considerable effort to attacking Ukip and its candidates.

All of this raises an obvious and intriguing question – why is none of this negative coverage reducing support for Farage and Ukip, which is quickly becoming the Teflon party of British politics? One possible reason is that some of Ukip’s financially struggling and left-behind core voters simply do not notice all the stories that excite Westminster village. The latest revelations rightly shocked the progressive and socially liberal, but they are unlikely to feature prominently on the radar of the average Ukip voter. We know from research, for example, that Ukip’s core voters do not tend to read the broadsheet newspapers that have run some of the most negative stories about Ukip and its leadership.

But clearly this is not the whole story. There is a deeper reason for Ukip’s Teflon status that is found in its roots of support.

As Rob Ford and I argued in our book, Revolt on the Right, Ukip is not fuelled by things that have happened since 2010. Rather, the party’s rise reflects strong differences in the values that guide different groups of voters in modern Britain. In short, since the 1970s there has been a deep and growing divide in the values that separate who we might loosely term pro-Ukip and anti-Ukip voters. Pro-Ukip voters are instinctively receptive to Farage’s anti-EU, anti-immigration and anti-Westminster message, and comprise between 25% and 30% of the overall electorate. These are the voters who grew up before Britain joined the EU, before increased immigration and in an era when genuinely competing ideological projects existed in politics.

But today these voters look at much of the contemporary media and see an institution that reflects everything they loathe about modern Britain: the seemingly endless celebration of rising ethnic and cultural diversity; a prioritisation of London over the rest of the country; a view of immigration that talks only about what it means for the national economy; a dismissal of working-class voters as bigots; and an unquestioning acceptance of Britain’s EU membership.

These are voters who like Farage but loathe Russell Brand. They have found Britain’s social and economic transformation profoundly unsettling and are watching the country evolve into a society they simply do not like. Now, the same tabloid newspapers that have spent more than 20 years telling these voters that they are right to feel anxious about immigration, the EU and politicians in Westminster, are telling them that they are wrong to support a party that is campaigning on the same message. Voters are not stupid. And, unsurprisingly, many of them have decided to give Ukip a free pass: “So what if that Farage has a few friends who are nutters,” they say. “He is the only one listening to us and standing up for our values.”

This is why the current response to Ukip does not come without risks. Progressives might celebrate as each candidate is exposed and ridiculed. Calling out racism where racism exists is important. But over the longer term this will not get our society very far. If it did, then Europe as a whole would not have seen a stubbornly persistent rise of radical right politics over a 30-year period.

Without turning attention to the deep and growing value divides that underpin Farage’s revolt, they risk inadvertently strengthening his support among voters who are quickly drawing a line from our media and political class to their unsettling experience of social and economic change.