Elections tend to be fought on two key fronts: the battle for the political narrative, fought in manifesto announcements and leaders’ addresses to the nation, as parties compete to set the territory and issues on which an election will be fought; and the battle for votes on the ground, fought in canvass returns and leaflet drops.

Last week’s byelection results in Essex and the north west show Nigel Farage’s Ukip on the march on both fronts. It did better than expected: in Clacton, Douglas Carswell increased the share of the vote he won as a Conservative in 2010 to be elected the country’s first Ukip MP; and in Heywood and Middleton, a 36% swing to Ukip slashed the Labour majority in one of its safe seats to just 617.

Such dramatic swings are unlikely to be replicated in the general election next May: the most Ukip could hope for would be a handful of seats. But to measure Ukip’s strength in seats alone would be to seriously underestimate its influence on the national political debate. For a political party that has only just won its first Westminster seat, it is exerting a remarkable pull: its brand of populism, built on a backbone of anti-immigration and anti-European sentiment, looks set to ensure these issues are centre stage in the election debate.

Ukip’s growing influence means it must now be subject to the scrutiny that it has so far managed to elude. The party’s 2010 manifesto included a swath of extreme proposals: banning football teams from having more than three foreign players in starting lineups, allowing mosques to be built only if approved in local referendums, banning the wearing of the burqa in public places, scrapping equality and discrimination legislation and “re-examining” inclusion in mainstream schools for children with learning disabilities.

Farage has sought to distance himself from this document, implausibly claiming he had limited influence over the party’s policy positions, despite his role at the time as the party’s chief spokesperson, MEP and former leader. These worst offenders have been stripped from the party’s current list of policies, which now represent a hotch-potch of left and right populism: anti-immigration, anti-Europe and anti-international development but also anti-NHS privatisation and anti-bedroom tax.

Yet while these policies are no longer to be found on paper, prominent Ukip candidates seem to have few qualms about expressing odious views about sexuality, gender, disability and race. Godfrey Bloom was forced to resign as an MEP after criticising aid being sent to “bongo bongo land” and describing female activists as sluts; Geoffrey Clark, a Ukip council candidate, published a manifesto calling for the compulsory abortion of foetuses with disabilities; and Winston McKenzie, the Ukip candidate in the 2012 Croydon North byelection, claimed that allowing children to be adopted by gay parents amounted to child abuse.

Perhaps most tellingly, Farage has been unable to hold himself back, declaring last week that people with HIV should not be allowed to move to Britain and arguing that people would be right to be concerned if Romanians moved in next door. The fact that he is a privately educated former banker has not seemed to undermine his credibility in attacking the establishment of which he is surely a part.

The main Westminster parties have much to answer for in explaining Ukip’s rise – and the lack of accountability it has faced. Historically, the fight for the political narrative has been in the hands of the two main parties. But the last few weeks have underlined the extent to which this is no longer the case. The common theme uniting both the Tory and Labour narratives is their negativity. Conservative political strategy has been relentlessly focused on proclaiming “you can’t trust Labour with the economy”. Cameron’s ill-fated Big Society project and the environmentalism that were both key planks of his modernisation project have sunk without trace, and his victory on gay marriage, hard-fought within his own party, looks unlikely to be strongly championed. Labour’s strategy, meanwhile, has been to paint the Conservatives as the defender of the wealthy few at the expense of hard-working families. While Miliband has tried to set out a new vision for Britain based on long-term reform of the economy, the party has struggled to communicate it to voters.

The failure of either party to produce a convincing positive vision for Britain has left space wide open for challengers championing a populist, anti-politics-as-usual narrative. In Scotland, this has been filled by the campaign for independence spearheaded by Alex Salmond. While critics of the yes campaign rightly highlighted gaping inconsistencies in the SNP’s proposals, and yes activists were accused of deploying bullying and intimidatory tactics, no one could deny that the SNP fought a campaign based on a positive vision for Scotland’s future.

In England, however, it is Ukip that has moved in to fill the gap. Its successes last week show that its central premise – blaming immigrants and European bureaucrats for the nation’s ills – has the potential to resonate strongly with the core vote of both main parties. They face a choice: they can take Ukip head-on, challenging its world view, or they can concede the battle to shape the territory of the next election, playing it on its own turf.

It is a mark of how brittle both Miliband and Cameron are feeling that they seem to be going for the latter. The Conservatives have promised an in-out EU referendum after the election that would dominate the first two years of the next parliament, and Labour looks set to further harden its immigration policy. Until recently, both leaders had been reluctant to criticise Farage too openly and both refused to condemn him as racist in the wake of his comments about Romanians, although Miliband did say they constituted a racial slur. The strongest reason being put forward by the main parties not to vote Ukip is not that their populism and scapegoating has nothing to offer Britain, or that they harbour candidates with odious views about race and sexuality, but that voting for Ukip might let the other lot in. This is a strategy that risks backfiring, further fuelling the anti-politics sentiment that is being so successfully stoked by Ukip.

It is a depressing indictment of the main parties that they appear to be allowing Ukip to dictate the turf on which the election is fought. Britain is not an illiberal or racist country: even a majority of Ukip voters support gay marriage. Growing support for Ukip is, instead, reflective of the fact that neither of the main parties has succeeded in building a convincing and positive vision for Britain in the wake of the financial crisis. Chasing Ukip’s tail on immigration and Europe is an implicit admission that they have given up on this most important of tasks.

BYELECTIONS