Newark, the Tories will hope, is Ukip's Stalingrad, the decisive moment when the purple tide is driven back and Nigel Farage's demoralised "People's Army" scatter to weep into their real ale in the nearest saloon bar. The Tories' temporary unity – better described as a "pre-election truce" – would have been at risk, had the result gone otherwise.

David Cameron's strategy of defusing the Ukip surge by promising an EU referendum and beefing up the anti-immigration mantra has merely played into Farage's hands, shifting political debate on to terrain where Ukip flourishes. Ukip would still have that precious commodity: momentum, the "big mo". Cameron would have been in big trouble and with limited room for manoeuvre.

It is easy to write the Ukip response: the fact that a 22% leap in our vote is portrayed as a failure just underlines what dizzying heights we have reached. This was number 258 on our target list, and lacked the older, non-university-educated, insecure working-class voters we depend on. Combined with the Tories – the most successful electoral force in the world – throwing every last resource at keeping the seat, we could never have won.

But Newark exposes why Ukip are doomed under first-past-the-post. Ukip is a protest vote, but we have now witnessed the birth of a protest vote against a protest vote. Byelections are supposed to be easy mid-term hits, and this one was called after a Tory MP had to resign because of the whiff of good old-fashioned sleaze.

Chris Bryant, the Labour shadow minister who led his party's Newark operation, talks of a complex "pentangle of switching": some Labour voters moved to Ukip to give Cameron a kicking, others who traditionally voted Lib Dem as an "up yours" to the political establishment have slipped over to the Farageists, and so on.

But many Labour and Lib Dem voters clearly voted tactically to keep Ukip out. Ukip's glass ceiling of support has been exposed.

Under our current electoral system, having a core vote which is spread disparately across the country is pointless from the perspective of winning seats. If you're a political geek, check out Electoral Calculus, a website which allows you to spend the afternoon typing in seat predictions at the next election. If Ukip wins 20% next year, the website points out, they are still unlikely to win a single seat.

Unlike even the besieged Lib Dems, they are unable to concentrate support in any given constituency.

The national polling explains why. Ukip are supposed to be a symptom of popular anger at the political elite, but – according to an ICM poll in March – they are the least liked and most disliked party. According to YouGov, while 28% of people had a positive view of Ukip back in 2009, it's now just 22%; and while 37% viewed them negatively five years ago, that has now jumped to 53%. The Ukip vote is dwarfed by the "can't stand Ukip" vote, and that is a formidable obstacle to them winning a seat.

This is what makes them different from the Lib Dems, their predecessors as the "None of the Above" party. Difficult though it now is to imagine – with Nick Clegg having gone from approaching the popularity of Winston Churchill to that of the ebola virus – pre-coalition Lib Dems did not have much of a stigma among the wider population.

That's why they excelled at winning byelections – that and cynically opportunistic campaigning which drives Labour and Tory activists doolally.

There is a debate raging on the left about how to deal with Ukip. Some believe that it is counterproductive to label Ukip – let alone its voters – as racist, lest it fuels a sense of the muesli-eating, sandal-wearing "liberal elite" closing ranks to shut down debate.

Others believe that failing to stand up to bigotry spewed by Ukippers allows the terms of political debate to be shifted, reversing the gains of the anti-racist movement as previously unacceptable bigotry returns to the mainstream.

But the evidence does suggest that the stigmatising of the Ukip brand has imposed a clear limit on its success. Ukip is an unsustainable coalition, too. Ukip's leaders are uber-Thatcherites, but Ukip voters tend to be economically leftwing people whose anger has been directed principally at immigrants.

That doesn't mean the Tories should crow too much. The near 9% swing against them in Newark is approaching the average by-election swing they've suffered since 2010. Crowing that Labour once held the seat is extremely disingenuous given boundary changes favourable to the Tories; and in byelections in Glasgow East and Glenrothes in 2008, the Tories suffered swings against them but still emerged the largest party in 2010.

Neither should Labour get comfy, given Ukip has shown it can chip away at their support. But the limits of Ukip are now abundantly clear; and when an opponent's frontier has been exposed, it is easier to start driving them back.