In the long slipstream of this year's party conference season, British politics seems to have gone strangely quiet. But listen closely, and under the sound of all that rain, you can make out something very interesting: the metaphorical forces politicians usually call "tectonic plates", shifting in ways that, three or four years ago, no one would have predicted. This winter's biggest political story, in fact, may turn out not to be focused on the Conservatives, Labour or the Lib Dems, but an organisation that until recently was routinely condemned to the fringes, or smirked about as a collection of eccentrics and oddballs.

But there it is: the UK Independence party, which has spent well over a year regularly scoring at least 6% or 7% in the polls, and often climbing as high as 11%, thus relegating the poor old Lib Dems to fourth place. At last week's Corby byelection, the party managed an impressive 14.3%, its highest-ever share of the poll in any such contest. That day, there was also a byelection in the seat of Cardiff South and Penarth, where it managed 6.1% – not nearly as convincing, but still its highest share in any Welsh election. And in the same day's somewhat shambolic elections for police and crime commissioners, Ukip's share of the vote per candidate once again put it ahead of Nick Clegg's lot.

This week sees the Rotherham byelection, where the party's prospects have been boosted by a remarkable story indeed: the local council deciding to remove three children from the foster care provided by a local couple who are Ukip members. The children are migrants from mainland Europe; Rotherham's director of children services, who was quickly condemned by both Labour and Tory politicians, said she had to be mindful of their "cultural and ethnic needs", in the context of Ukip's policies on multiculturalism. Ukip's website now features the slogan "All roads lead to Rotherham", the stylised image of a family of five, and a headline about what it calls the "Ukip foster care uproar".

Yesterday, the Tory MP and party vice-chairman Michael Fabricant published a report titled The Pact, in which he advocates an electoral deal between the Conservatives and Ukip, on the basis of a referendum on Britain's EU membership, and a place in a future Tory cabinet for the Ukip leader Nigel Farage.

Fabricant – who on Sunday night was reported to be having "social drinks" with David Cameron – reckons that the ongoing battle between the two parties cost his side as many as 40 seats (and, therefore, an outright majority) at the last general election. The Tory leadership duly poured cold water on his suggestion, but the underlying thinking was hardly revelatory: Ukip's rise is jangling Tory nerves, and with good reason. On Monday, Farage talked about the possible game-changing effects of someone "grownup and sensible like Michael Gove" becoming leader of the Tories: the aim, one suspected, probably had more to do with mischief than constructive politics.

Ukip already has 12 members of the European parliament, including Roger Helmer, who was elected as a Conservative, but jumped ship in March this year. There are three other ex-Tory Ukipers in the House of Lords: Lord Pearson of Rannoch, the 21st Baron Willoughby De Broke, and Baron Stevens of Ludgate. While we're here, it is also worth noting the sole Ukip representative in the Northern Ireland Assembly, David McNarry, a former member of the Ulster Unionist party – and the party's presence in local government. Ukip now has 158 people serving on local councils, though the vast majority are concentrated at town and parish levels, a number regularly swelled by more revolting Tories.

They are all committed to a self-styled "libertarian, non-racist party seeking Britain's withdrawal from the EU", whose ideas are built on the claim that even the Conservatives – and read this bit slowly – "are now Social Democrats", and that the main parties "offer voters no real choice".

Aside from pulling out of Europe, Ukip's other notable positions and policies seem purposely designed to cut across what remains of the metropolitan "modernisation" agenda that Cameron and his supporters brought to modern Tory politics. Chief among them is the belief that climate change is a matter of debate and "wind power is futile", the contention that there should be "real and rigorous cuts in foreign aid" (to be "replaced with free trade", apparently), and support for grammar schools. Given half a chance, Ukip would also freeze "permanent immigration" for five years.

The party's prevailing tilt is in the small-state, cut-spending direction, though it would hold on to Britain's nuclear weapons, and "make increased defence spending a clear priority". It is opposed to gay marriage (though it's OK with civil partnerships), and advocates an end to the ban on smoking in "allocated rooms in public houses, clubs and hotels." The party's radical rightwing credentials are also flagged up by its avowed belief in a flat rate of income tax, an idea that has found favour in Serbia, Ukraine, Slovakia, Georgia, and Romania.

In 2006, much to Ukip's fury, Cameron famously called them a party of "fruitcakes, loonies and closet racists" (weirdly, over the weekend, the Downing Street press office seemed to retract at least the third of these suggestions, only to un-retract it). There have been occasional reports about Ukip members with links to the far right, and the BNP in particular (though on this score, to be fair, the party is vigilant). There also is a low hum of online noise about the party's associations with other political parties in Europe, the kind routinely described using such terms as "ultra-nationalist" or "socially authoritarian": in the European parliament, their MEPs are part of a grouping called Europe of Freedom and Democracy, which also includes the Italian Northern League, the Lithuanian Order and Justice party, and an outfit from Greece called Popular Orthodox Rally. In 2009, Ukip peer Lord Pearson invited the Dutch politician Geert Wilders to the House of Lords, where he showed a film titled Fitna, linking the content of the Qur'an to terrorism.

Still, if you like the cut of Ukip's jib, you might like to think of its members as bold trailblazers for the future of the radical right. If you are being slightly less generous, you might agree with the verdict of an internal Tory document that called them "cranks, gadflies and extremists". Neither view, though, answers this year's most pressing question: why has the party's support suddenly ballooned?

According to John Curtice, the renowned psephologist and professor of politics at Strathclyde University, the answer is inevitably bound up with two institutions that have each had a grim 2012: the European Union, and the British Conservative party.

"The simple answer is that the public are getting much more Eurosceptic," says. "And yes, the public is pretty Eurosceptic, but it's not clear that it's any more Eurosceptic than it was in the late 70s and early 80s. The other argument is, you've got a bunch of people out there who are normally Tory supporters, and they're not entirely sure that Cameron's got it, they maybe think that Osborne has made too many mistakes … they've lost confidence in the competence of the Tories. Now, if you're in that situation and you're a voter on the centre-right, where are you going to go?

"You're not going to vote for the Greens. You can't vote for the Lib Dems. You think the BNP is going too far. The answer might be Ukip, because quite a lot of their policies are quite similar to the Tories. And you perhaps think they'd at least do something about European immigration, which none of the other parties would." According to Curtice's numbers, around 7% of the people who voted Conservative at the last general election would now vote Ukip; he does not quite concur with Fabricant's belief that Ukip could cost the Tories up to 40 seats at the next election, but if they even threaten 20, "that's still non-trivial".

In 1991, an LSE historian and academic called Alan Sked formed the Anti-Federalist League, a group-cum-party opposed to the Treaty of Maastricht, the agreement that formally established what we now know as the European Union. Two years later, it became the UK Independence party. In 1995, it held its first national conference, which drew 500 people. At the general election of 1997, it was rather overshadowed by the late Sir James Goldsmith's Eurosceptic Referendum party, though with his death later that year, Ukip quickly found itself at the forefront of non-Tory Euroscepticism.

Sked, however, soon left, claiming that he had begun to fear that people who did not share what he calls "liberal British values" were joining the party in ever-increasing numbers. He now claims that once he exited Ukip, text stating that the party had "no prejudices of any kind against any lawful minority" disappeared from its membership forms.

In 1999, Ukip got its first three MEPs. Five years later, it reached its first watershed moment, when 12 were elected. But around this time, Ukip fell victim to the revived ambitions of the Labour MP-turned-talk-show host Robert Kilroy-Silk – who fancied becoming leader, until his aims came to nothing, and he left to found the long-forgotten party (the French would call it a groupuscule) Veritas. The way was thus opened for the rise of Nigel Farage, a commodity broker and former Tory who became Ukip leader in September 2006, although he resigned three years later, to concentrate on his efforts to become the MP for Buckingham.

There, he was cocking a snook at political convention by running against John Bercow, the (nominal) Tory and speaker of the House of Commons. But in the event, Farage came third, behind an independent called John Stevens, who campaigned with the aid of a character called Flipper the Dolphin – though that failure was eclipsed by one of the most remarkable moments of the 2010 campaign, when Farage crawled from the wreckage of a light aircraft after a Ukip banner got wrapped around its tail fin (weirdly, the pilot was later found guilty of making death threats against him, in a separate incident).

In 2009, it had been revealed that Farage had taken £2m of EU expenses and allowances, which he claimed had been used to promote Ukip's message. Within his party, the story obviously did him no harm at all: in November 2010, he once again became Ukip's leader, and is now a firmly embedded part of the culture – an apparently unembarrassable, foghorn-voiced operator (some have likened his tones to those of Zippy from the 70s children's TV show Rainbow) who proudly smokes and enjoys a lunchtime pint of bitter, and who characterises his relations with the Tories as a matter of "war".

For at least one of his old colleagues, however, Farage's success is less important than the rightwing politics that he has firmly planted in Ukip's collective soul. "Ukip is far too rightwing for me," says Sked, from his home in the Scottish Highlands. "And they've gone native: they're mainly interested in their seats in the European parliament, and their pensions and allowances. All the energy seems to be directed to keeping them in Brussels and remaining well-paid members of the European parliament. It's gone askew."

Once Sked gets going, there's no stopping him. "Their other obsessions seem to be anti-Islamic and anti-immigrant," he says. "I couldn't believe what happened at the last election. The country was facing its greatest economic crisis since the 1930s, and all Ukip could say was 'Ban the burqa'."

The party's high-ups, of course, are having none of that. Paul Nuttall, 35, is a Liverpudlian former academic who joined Ukip after a spell living in Spain ("I saw the failure of the euro firsthand," he tells me), became an MEP for the north-west region, and is now the party's deputy leader. He puts their apparent surge down to "being proved right on everything to do with the European Union", and the endless warnings the party has dispensed about "mass, uncontrolled immigration". He also reckons that Ukip "has become professionalised. It's moved on from being a single-issue pressure group. We've developed a whole raft of domestic policies that people find attractive."

In the European elections of 2014, he reminds me, the party's aim is to finish first; at the next year's general election, they want nothing less than a "political earthquake", though what that might mean remains unclear. But why not, I wonder, swallow hard and get with the Fabricant programme? A deal with the Tories, after all, would guarantee them at least one seat in cabinet – and, one assumes, a handful of MPs.

"The biggest stumbling block at the moment is the prime minister himself," says Nuttall. "He can't be trusted on the European Union. And he's described us as closet racists in the past: he had the opportunity to retract that, and then he retracted the retraction."

Sked's accusation that Ukip has long since fallen in love with the perks and privileges of Brussels, he tells me, is "nonsense", and the idea that the party is "obsessed" with Islam is also given short shrift. "I don't think we've talked about the burqa since 2010, to be perfectly honest with you. It's not something that's a lead policy of ours. But I will say: if I can't walk into a bank with a crash helmet on, then I think the burqa should be removed."

Could a devout Muslim be a wholehearted supporter of Ukip?

"I don't see why not."

We end our conversation with his party's rum assortment of allies in the European parliament, and another chance to rummage through more arcane rightwing parties that do their thing in Brussels: among them, Helsinki's own True Finns, and the United Poland party.

"Groups in the European parliament form as a marriage of convenience," he says. "We don't really deal with each other's domestic policies … I'm sure if you look at the Conservative and Labour parties, they have people in their groups who they wouldn't necessarily form pacts with in this country. It's just the nature of the beast."

Even if they're on the far right of politics?

"I'm sure there are people you could say were on the far right of politics in the Conservatives' group, and people on the extreme left in the Labour party's. You just have to hold your nose."

His last sentence sounds like the kind of thing that you can get away with on the fringes, but that a new life at the centre of the action might make that a bit more difficult: "Their domestic policies are nothing to do with us."