Five years ago, anyone interested in political predictions would probably not have consulted Nigel Farage. Even five months ago he would have been a long way down most people's lists of plausible Cassandras. But if you want to know what our political future might look like, Ukip's party leader is becoming harder to ignore.

"Five years from now, Ukip will have changed the face of British politics," Farage says. It might become "a huge political party" – or it might merge with another – "a realignment in British politics, a sort of SDP moment on the centre right". Alternatively, a new Tory leader might have ousted David Cameron and embraced Ukip's agenda. "I've no idea which of those three scenarios will play out," he admits cheerfully, nor does he have a favourite. "I don't care. What I care about is that we achieve our goals."

Farage turns out to be one of the most surprising politicians I have met – charismatic, funny, indefatigably good-natured and essentially cheerful towards absolutely everyone, apart from the prime minister and Rotherham council. He is dressed for a home counties golf club AGM, and smokes with the committed defiance of a dissident setting himself on fire; when I confess to having given up, he offers a doubtful, "Probably wise", as if I had outed myself as a raw diet fruitarian. He has tried to quit a few times, and even lasted three months once. "But who wants to live like that? We're only here once, you know."

His facial expressions are like a cabaret all of their own – wildly camp, part Frankie Howerd with a hint of Larry Grayson, defying both symmetry and gravity. But he listens carefully to questions, and even seems to want to answer truthfully. Occasionally he opts for strategic ambiguity or dodges a question – but with heroic reluctance. When he tells me he has no personal ambition for high office, and cares only about seeing Ukip policy implemented, I believe him. Whether or not he is also right about Ukip rewriting our political future is another matter – but so far he has been right about more than most liberals would like to admit.

After the recent dramatic poll surge and byelection triumphs, senior Conservatives are openly discussing an electoral pact, in which Ukip would get a straight in-out referendum on EU membership in return for not standing against Eurosceptic Tories. It is a measure of Farage's bombastic confidence that he acts as if the very idea of dealing with David Cameron is beneath him – "There's no way we could work with that man under any circumstances" – and suggests a pact with Labour might be on the cards instead, "if Jon Cruddas's view prevails within the party". Farage has never forgiven Cameron for calling Ukip members closet racists, but, on the other hand, has to credit him with making Ukip so popular.

"From a purely personal Ukip perspective I'd say thank God for David Cameron, wouldn't I? Who has told a whole load of lies, quite deliberately, before the election to shore up the Tory vote – going to repeal the hunting ban, get rid of the human rights act, reduce immigration. Everyone said to me, 'Well, Nigel, I like what you say, but just you wait until David gets in, he's playing a very clever game but once he's in he'll be really tough.' Well, David is in."

The Lib Dem coalition partners have not watered Cameron down, Farage maintains, but merely provided an alibi for his wetness. "It would have been easier for us had there been a Tory majority. It would have been clearer to everybody that Cameron is a social democrat."

His mobile rings, and he answers. "Yeah … mmhm … mmhm … yeah ... interesting. Right, interesting. Thank you." That was very cryptic, I say, when he hangs up. "Yes, very interesting." About? "Cameron's speech." Who was that? "Umm, a Tory." How often do they call? "All the time."

Farage is one of the few politicians who looks convincingly appalled when invited to imagine waking up one day as PM. "I'd have a hell of a headache, wouldn't I?" he laughs, adding briskly, "But it's not going to happen. No, I'm a full-time campaigner. Different psychology." The possibility of becoming deputy PM, with Ukip replacing the Lib Dems in a new coalition, has, however, clearly crossed his mind. "That isn't my ambition," he insists. But is it an attractive proposition? He spreads his arms, his face a frenzy of innocence and insinuation. "I don't know, I don't know. I can't really answer it."

That's such a proper politician's answer, I say. "Oh God, I'll be becoming respectable now, and then it'll all be over," he groans. He has, he concedes, learned that "If I have a fault, it's that I get a bit carried away." He laughs. "I do believe in things quite strongly and I can get a little bit excited sometimes."

Most of the people now joining Ukip are not motivated chiefly by Europe, and if the UK were to pull out of the EU that would only be phase one, as far as Farage is concerned. "Leaving the EU doesn't solve all our problems, it just gives us back the ability to solve them. We are a very important catalyst for change in the national debate on a wide range of subjects and a completely alternative view on the size and role of government." Once you get beyond Europe, though, much of Farage's political landscape looks very shaky.

He won a lot of Tory support by opposing gay marriage, chiefly because it was an affront to religious values. "Tolerance is a two-way street, and the whole equality rights agenda has come to the point of head-on conflict with religious faith," he declares, as if such a conflict must de facto discredit the equality agenda. It takes some nerve to oppose gay marriage on religious grounds – while adding, "I know the Anglican church isn't much good, but mind you, with that idiot having run the show for the last 10 years that's hardly surprising. Couldn't even clip his beard for the royal wedding!" – when, on closer questioning, it transpires Farage isn't even really a Christian. He claims never to have thought about whether he will go to heaven, or even if such a place exists. "Never." He goes to church four or five times a year, and thinks it plays "an important role in our society", but as for believing in God, "I think there is something there, but that's as far as it goes." It sounds to me as if he's agnostic. "Well you'll have to draw your own conclusion," he says, looking slightly embarrassed.

Nigel Farage worries that if drink-driving laws are harmonised across the EU: 'Anybody who goes to a rural pub on a Sunday lunchtime will be in breach of the law.' Photograph: Peter Macdiarmid/Getty Images

Like almost every rural Tory I know, he claims to care passionately about the environment while querying both the fact of climate change and the culpability of CO 2 emissions. When I ask about our drink-driving laws, he reluctantly offers: "I suppose I support them. I suppose I do." He has no sympathy for anyone caught drink-driving in London. "But if you're caught out in the rural parts of the country …" But if the legal alcohol limit is correct in the city, then anyone who exceeds it is a danger wherever they are. "I just don't like what's happening in rural England, really. Pubs, clubs, social life closing down." And if drink-driving laws are harmonised across the EU, he adds indignantly, lowering the legal alcohol limit even further: "Anybody who goes to a rural pub on a Sunday lunchtime will be in breach of the law."

But his party's enthusiastic libertarianism goes out of the window when it comes to a pleasure its core members aren't so keen on – illegal drugs. Farage's own instinct would be for wholesale decriminalisation – which would almost certainly broaden Ukip's appeal among younger urban voters – but the policy isn't even up for debate. "It would be completely impossible for me to win that debate within the party. And a general doesn't try to fight every battle."

If Farage has perfected one political trapeze act above all others, it is friendly gratitude for a movement he concedes was basically bonkers when he first joined. Now 48, Farage had never planned to be a politician. A south London public schoolboy, the son of a stockbroker, his youthful ambition was uncomplicated: "Be rich, absolutely." Heading straight for the City at 18, he was earning £200,000 by 21. "That was a lot of money, wasn't it?" But then the UK joined the ERM, "and I just thought it was cretinous." Having started his own business, he forgot to spend the 90s getting rich as he devoted himself to Ukip, culminating in unlikely triumph in 1999 when he was elected to the European parliament. But those were thankless, wilderness years – surely he must have wondered if the whole enterprise was mad? "Um, I didn't think the concept was mad. I thought the people, in many cases," and he starts to laugh, "were not to my taste.

"Ukip in the 1990s, the people in it and who voted for it were in the main 'Disgusted of Tunbridge Wells'. I mean, you look down the membership list in 1994, anyone below a half colonel was a nobody," he hoots."I used to say you could always tell it was a Ukip meeting by the number of bomber command ties in the room. It was that generation." Was it his milieu? "No! I was the odd one out. Which I loved, of course. I've always liked to be the odd one out, wherever I am."

He gave everything he'd got to the 1999 European Parliament electoral campaign. "I was so knackered I had a tic in both eyes." He knew the path of an anti-EU MEP would never be easy, but hadn't bargained on his own colleagues creating his party's biggest difficulties. "In those days in Ukip there were an awful lot of people who didn't pass the Farage test. Number one, would I employ them? Do I think they could do something competent, professional, rational? Number two, would I like to go to the pub with them? You haven't got to pass both. But no, these people didn't pass either."

Worse, they were putting off potential new members. "Very competent people would come to a Ukip meeting and go: 'Nah, not for me.'" Farage freely admits some of his own personal recruits turned out to be calamitous misjudgments of character. When I mention Robert Kilroy-Silk's brief, disastrous spell in the party, Farage performs a little pantomime of horror at the memory.

"He is a monster, he really is," he chuckles, shuddering. "But then, do I want a party where we've got some eccentrics and occasionally someone causes us an embarrassment – or do I want a party made up of a bland lot of ghastly people whose names I don't even know? I'm in politics, and I've never heard of half the MPs who come on the telly. I meet them at cocktail parties and after 30 minutes I can't remember their name!" He hoots with laughter. "And they have absolutely no connection with the real world. No, I want Ukip to be a party of free-thinkers."

When a Ukip council election candidate made headlines by proposing compulsory abortion for every foetus with Down's syndrome, Farage admits "it was a bit of a shock", but is keen to point out that while he leads the party, he does not micromanage it. He is, however, proud of a unique condition he has imposed on membership; "We are the only UK political party that bars you from becoming a member, let alone an officer or candidate, if you have ever been involved with the BNP." Critics might say his is the only party that needs to. "No, we've seen defections from the BNP to Labour in the north of England, so I won't take that. People come to me and say, 'Well this is how I feel, Nige,' and I say, 'That's great, you're not for us, go and join the BNP, cheerio.' I think the message loud and clear is that if that's how you view humanity, we're not the right people for you. And I think it's working."

In fact, he has his eye on the West Indian community. "That's one where Ukip could do very well. They love our pro-Commonwealth stuff." Mostly, though, he is chasing C2 votes. "Very much so. The biggest group for us are what I would call not particularly well off but aspirational people." However, his biggest obstacle in winning over any voter remains tribal loyalty. If the nation voted on manifestos alone, "Ukip's support would be 25-30%."

As for the ongoing Eurozone crisis, I'd expected him to pretend he's not secretly rejoicing at each new disaster, but hadn't expected the protestations to sound sincere. "No, it's desperately sad. Inevitably, this is going to end in some kind of widespread violence, the likes of which none of us wants to see. We're not going to be digging holes in Belgium, facing each other, no. But will we get terrorist groups emerging in Greece, Spain and Portugal? Yes."

Farage seems to have an uncanny knack for conjuring capital from crises. He nearly lost his life in a plane crash on election day in 2010, which left him with permanent damage to his back – but even his injury has turned out to be rather helpful. A famously enthusiastic drinker: "Before, I would always have been, at any event, the last to leave – kicked out by the cleaners. That isn't the case any more." That's probably been good for his political career? "Well it might make me a bit more fit for duty, yeah."

As for the most unlikely bit of recent sensational political luck – the fostering row in Rotherham – he is in no hurry to let the matter lie. "We won't let this rest, no, no, no, no, no, that big bully-boy council, if they think they can brush me aside, they've got another think coming. They've not done a good job for kids in Rotherham."

But a brilliant job for Farage?

"Ironically, yes," he beams. "Funny how life works out, isn't it?"