The room was packed in anticipation of good debate. But from the start of the Conservative conference fringe event on immigration there was an air of unpleasant hostility from a group of older people – as I suggested the Tory party needed to adapt to the modern world, adopt a more rational approach to migration and reach out to younger voters and those from ethnic minorities.

One white-haired man yelled that I was a prat when I praised the global success of British universities. Another elderly woman called me stupid. A third person said I was pathetic.

Afterwards, an activist told me of a friend who had bumped into a woman in a burka, then been abused. She asked what could be done about “them”, referring to fellow British citizens her party should be aiming to win over.

Perhaps I was too confrontational over their prejudices, pointing out the hypocrisy of Britons who feel they have a God-given right to move anywhere on the planet but demand the drawbridge be pulled up over the channel. My fellow speakers, Andrew Green of Migration Watch and a thoughtful Liam Fox, proved more to the taste of these particular Tories – although others quietly urged me afterwards to keep fighting for progressive values.

The event underlined the strange mood in both Birmingham and British politics. Last week Labour, heading towards a hesitant return to power with the support of just over a third of voters, seemed subdued under a leader whose bungled speech underlined concerns over his abilities. This week the Tories seem similarly downbeat, despite the growing strength of the economy, and having seen their set-piece rocked by defections and an old-fashioned sex scandal.

At the root of this unease lies the Ukip insurgency, with the party gaining members and MPs with its cheapjack appeal to older, white, working class people perplexed by rapid social change and pessimistic over their future. A former City banker and lawyer is the latest Tory turncoat posing as their champion – and Downing Street fears a third defection this week to devastate coverage of David Cameron’s speech. “Too many of the nutters aren’t returning our calls,” one insider said gloomily.

Cameron gets little credit for the tough task of governing through challenging economic and global times while holding together his fractious coalition. It is essentially a three-party coalition between the majority mainstream Conservatives, the hard right obsessed over Europe and immigration and the floundering Liberal Democrats. There have been some good reforms – schools, gay marriage, planning and pensions – alongside economic success. For all the furore over migrants, the humdrum issue of pensions is far higher up the list of voters’ personal concerns.

But coalition has cursed the Tories, just as it has the dissolving Lib Dems, especially under such a pragmatic prime minister. The rabid right still claim modernisation failed in opposition as they subvert their party in office. Yet the Tories were way ahead in opinion polls when they promoted a communal conservatism that focused on schools, hospitals, poverty and the environment – until forced by economic downturn to embrace austerity. Even then, with talk of cuts highlighting voters’ fears over public services, they gained more seats than at any election since 1931.

But in power the Tories allowed the Lib Dems to portray them as the nasty party, then compounded this by tacking to the right in a vain attempt to appease hardline ideologues who can never be mollified. This flip-flopping fuelled the mistrust of politicians being exploited by Ukip – and a moderate prime minister is now seen as significantly to the right of the average Tory voter.

The failure to learn the lessons of the past by banging on endlessly about benefits, Europe and immigration is astonishing. There needs to be more, not less, modernisation. Instead, the Tories focus fruitlessly on these fearful older voters largely lost to Ukip, an inevitably declining sector of the electorate, while reinforcing an image that drives away the younger, female and ethnic minority voters needed to survive and thrive as a political force.

Ultimately, the question is not why are these MPs defecting, but why do politicians with such divergent views stick together? Perhaps politics is going through a process of disruption similar to that driven by technology in almost every other aspect of life. It does seem absurd to expect our tired model of binary party politics to endure in a time of transparency, with all that tedious tribalism and parroting of lines.

In the short term, the Tories must decide either to offer an optimistic vision of the future or just pander to the pessimists in a probably doomed bid to win the election.

Beyond that, it is hard not to wonder if these divisions need to be resolved with a cathartic full-blown split, as with Labour in the early 1980s – although this time it would be the militant tendency on the flank shearing off. As always in politics, there are egos and personal vanities in play. Yet what really binds the many decent and tolerant conservatives to those misanthropes filled with fear and rage against modernity?