In November 1991, an advert appeared in the Evening Standard inviting interested parties to a meeting of the Campaign for an Independent Britain at Westminster Central Hall, a short walk from the Houses of Parliament. Among the few hundred attendees was a 29-year-old commodities trader called Nigel Farage. He was impressed, but he felt he was too busy to really get involved. So he wrote the organisers a cheque for £50 and, as he puts it in his autobiography, "popped it in the post on the way to the pub".

Nearly 22 years later, Nigel Farage was back at Westminster Central Hall, but this time, he was a rather more central figure. The Campaign for an Independent Britain ultimately spawned Ukip, and Farage had somehow become its leader. With the party edging ahead of the Liberal Democrats in the polls and drawing more support than ever from disgruntled Conservatives, the conference, he explained later, was the most important in the party's history. "It was the most delegates we'd ever had," he said. "And there was a very marked change in the type of person there. The big idea was to launch new people and for me... not to take a back seat, exactly, but maybe not to emerge quite as dominant."

Farage's speech was scheduled for Friday morning. In the minutes before he entered the hall, a crowd of photographers and cameramen jostled for position at the edge of the stage, and there was a sharp change in the atmosphere. He walked in to a thumping and incongruously continental techno accompaniment, waved to the whooping delegates, and checked that his ancient Nokia was switched off. The speech that followed was warmly received by its audience, the only downside being a Nixonian sweat under the lights that made Farage, oleaginous at the best of times, look as if he'd just got out of the bath.

Download the new Independent Premium app Sharing the full story, not just the headlines

One of the loudest cheers came for Godfrey Bloom, the MEP for Yorkshire and the Humber, who had recently caused something of a stir with a derisive reference to "bongo-bongo land". "I had the most blistering row with Godfrey in a Strasbourg restaurant the other day," an indulgent Farage told his audience. "He wants to fight for his beliefs and I was saying that we need to stick to the big messages." But, he added, "be in no doubt: we must be the party of radical alternatives and free speech."

Farage left the venue and set off on a publicity blitz. A bit later, travelling between studios, he switched on the radio to hear the one o'clock news. His speech was the first item. Thinking it over in Ukip's headquarters a couple of weeks on, a wistful look came over his face. "I thought, yeah, I've nailed this," he said, snapping his fingers for emphasis. "And then it all went wrong."

As Farage crisscrossed the city, Bloom was at the Cinnamon Club restaurant, speaking to a fringe meeting on women in politics. In an ill-considered joke, Bloom called some of the women present "sluts". He then sailed outside to be confronted by a group of reporters. One of the group kept calling him Mr Godfrey, a perfectly calculated tactic to wind him up. As he strode away, he was pursued by Channel 4 News's Michael Crick, who he proceeded to thwack over the head with a copy of the conference programme. > The ensuing media furore was immediate, gleeful, and extensive, and by the teatime news, Nigel Farage's speech was very far from the top of the bulletin.

It was a crisis that summarised Ukip's controlling paradox: the closer their idiosyncratic brand of anti-politics gets them to the promised land of real influence, the more obstinately it stands in their way. Certainly, as he surveyed the wreckage a fortnight later, Farage had reason to reflect ruefully on his paean to the "party of free speech". "Of course I was annoyed with Godfrey," he said. "We've had a whole summer of these distractions. What I had told him at that restaurant was that we've got to focus."

The problem, I suggested, is that by insisting on that sort of discipline, you risk alienating the very people who have brought you the success you've had so far. "Look," said Farage, drumming his fingers impatiently on the table as he spoke.

"It comes down to this. What is Ukip for? What is it for? There are a lot of people in Ukip who think that it's for saying things, for providing opposition, for providing an alternative narrative, maybe shocking a little bit. I've done plenty of that over the years. But New Ukip, if I dare to call it that, is about fighting elections. It is about putting together election-winning machines. That's what I'm trying to bring."

But this approach is not always popular. "This is the third most popular party in Britain now," complains one insider, "and yet it's incredibly centralised, it's run from an inner core of Nigel and his sycophants, Nigelistas around him, people who've been given advances and promotions and jobs, and it's very unhealthy how much control he's got for a democratic party. He doesn't wield the knife himself, he's always got others to do it for him. The danger for Nigel is he has some very keen people around him for whom he can do no wrong."

Inside Politics newsletter The latest news on Brexit, politics and beyond direct to your inbox Enter your email address Continue Continue Please enter an email address Email address is invalid Fill out this field Email address is invalid Email already exists. Log in to update your newsletter preferences Register with your social account or click here to log in I would like to receive updates on politics every morning by email Update newsletter preferences

By the end of Friday's conference, Bloom had lost the party whip – not at Farage's instigation, he insisted, but at that of the party chairman. However much of a mess the day had been, Farage took the optimistic view that it had not been entirely unhelpful. "On a bad day, Ukip was a bit like the rugby club coach on the annual trip to the seaside," he said. "Fun though that is, I think that image probably has been left behind. The way it was handled, it may well be looked back upon as the moment that Ukip decided to take life a bit more seriously."

Such an assertion, his supporters say, would have been anathema to the 29-year-old who wrote that cheque and then set out for another night on the piss. The standard narrative of Farage is of a reluctant public figure, a straightforward sort of chap forced into politics by patriotism. "I'm sure he longs for the freedoms of the old days," says Mark Daniel, a novelist and former Ukip media officer who has known Farage for more than a decade. "But he also recognises that if he is to attain what he set out to attain, this is what it takes."

Farage has just finished three years in his second stint as leader. He is a veteran, and if he is to have a time, it must be now. There is no room for error any more: the success or failure of his bid to wrangle the party into some sort of order, and the potentially apocalyptic consequences for the Conservatives if he does, will be central to the outcome of the next general election. Even if he can't become an MP, he can set the agenda for the Tory party, in itself a glittering prize. And yet the lure of the old days – and the abiding strangeness of his party – are very hard to escape. When he finishes a day's work at the European Parliament in Brussels, he goes back to a flat that he still shares with his former Defence Spokesman: Godfrey Bloom.

Nigel Farage has nearly died at least three times. The first, according to his boisterous autobiography Flying Free, was in 1985, when he was a 21-year-old broker on the London Metal Exchange. Walking home from the station after a heavy night's drinking, he was hit at a pelican crossing by a VW Beetle, and left with broken ribs, a leg that almost had to be amputated, tinnitus, and a hangover that had his mouth "feeling like Queen Nefertiti's gusset". On the plus side, he fancied one of the nurses, Clare Hayes, who would become his first wife, and with whom he would have two sons.

Even after a second brush with the eternal, a diagnosis of testicular cancer a few months later, he quickly bounced back. "The first time I met my oncologist, in this hospital bed, I had the racing on, and a phone line open to a bookmaker, and I had a fag on the go and a glass of something and I'd had the operation [to remove the cancer] 24 hours before. And he came in and sort of stared at me, and he said, well, Mr Farage, after an experience like this, many of my patients spend the rest of their life drinking carrot juice, and some go the other way. And I suspect you're a member of the latter category." He grins at me and lets out his surprisingly endearing laugh, a sort of spluttering hiss that sounds like a steam train leaving the station.

To a journalist from what Farage sees as a hostile mainstream media – albeit a segment of it that he is shortly to join, with a column in The Independent from next week – there's an interesting challenge to this warmth: his friendliness seems to bristle against the ridicule to which his party has so often been exposed. When I ask him about one of the more controversial planks of his speech, a diatribe against an anticipated influx of Romanian and Bulgarian immigrants that was predicated on dubious statistics about those who are already here, he counters brightly: "We're not what you expect! We're not what you want! 1950s Britain, albeit with the stocks back on the green – that's what you expect to find. But it's not what you find."

The cancer anecdote is, in a curious way, similarly defensive. Likeable though he is in the telling, like most of Farage's stories about himself, this one comes with a defiant swagger: it makes a fetish of the stiff upper lip, and in doing so acts as an implicit rebuke to the perceived feebleness of the modern mainstream, the Elf'n'Safety PC Brigade who want to straighten our bananas.

This brisk sensibility is hard-wired. Consider, for example, Farage's view of the effect of the early exit of his father, Guy, on his childhood. "People get divorced," he says shortly. (He did so himself, and is now married to his second wife, Kirsten Mehr, with whom he has two daughters; their union has survived a News of the World report, strenuously denied, that he had sex seven times in one night with a Latvian woman called Liga who he met in a pub in Biggin Hill.) "That's the way life is. I suppose I think we can over-analyse some of these things."

The other side of this trenchant outlook is a knee-jerk resistance to conventional wisdom. This, too, was visible in Farage from his school days, when he became a devotee of Enoch Powell and – according to a contemporaneous letter unearthed by the vexing Michael Crick – was considered a "racist" and "neo-fascist" by some of the teaching staff. Farage, for his part, says that while he was a "troublemaker", he was certainly no racist.

It is, perhaps, unsurprising that he dispensed with university altogether, instead leaping straight into a job in the City. "I couldn't wait to get cracking," he says. "I don't want to be a scruffy student, I want to be out there." He seems to have had a fine time as a broker, notwithstanding the occasion he lost a silver warrant worth £90,000, and to have invented himself in a Thatcherite world that suited his attitude to life down to a tee. "To me, he was a Tory," says David Campbell Bannerman, a former Ukip deputy leader who has now defected to the Conservatives. "He always comes across as a Tory – a City man, very energetic." (Indeed, several people I spoke to saw him as a natural Tory backbencher.)

Despite his hesitance about the Anti-Federalist League, Farage was soon on board with Ukip, and widely recognised as one of the party's best public speakers. But, he insists, he had no designs on the top job. "I was always a reluctant leader, really," he says. "I had no intention of ever getting involved in politics. I saw myself making a lot of money, and certainly by this age being retired."

The question of when that started to change is a fraught one. For most politicians, of course, ambition would be an acknowledged part of the package; but Farage's personal brand is predicated on not being like other politicians. He himself suggests that both of those spells as leader have been for want of a better alternative, and certainly Ukip's senior figures have not shone brightly in the past. Mark Daniel, who wrote a history of Ukip and is thanked effusively in the acknowledgements of Farage's autobiography, says that the standard of candidates "worried me considerably, because there didn't seem to be anyone who was capable. I think Nigel would have liked to find others to step into his shoes."

But another version of events casts Farage as far more Machiavellian than he would admit. "He was always the power behind the throne," says one insider. "He just preferred to pull the strings." And Nikki Sinclaire, a long-time party official who became an MEP in 2009, scoffs at the idea that he was dragooned into office. "It's absolute rubbish," she says. "He wanted it, but he didn't want it too early. He wanted to be seen as the peacemaker who was getting on with the job. He only wants to be leader when he thinks the party can be successful."

One piece of evidence for this theory comes in that curious split in Farage's terms in office. He stood down as leader, the official version goes, in order to better concentrate his energies on his own doomed general election battle in 2010 against the House of Commons Speaker, John Bercow.

A more sceptical analyst might say that he saw that the party's chances of doing well were small, and consequently moved to absolve himself of responsibility for the failure, thereby setting up a glorious second coming. "You do get the feeling that he is there for his own good," says Craig Mackinlay, another former Ukip deputy leader who has defected to the Conservatives. "Ukip is only there for the convenience of Nigel and very little else." David Campbell Bannerman, likewise, says that there is "a bit of the showman to Nigel. He likes to be asked back, for his people to beg him to return".

Then there is the suggestion that Farage's leadership style has failed to adapt to the party's newly broader base of talent. Another disgruntled MEP who has left the party, Marta Andreasen, says that he can be "autocratic", and "personally ruthless with people". One example, Nikki Sinclaire says, can be found as far back as 2004, when Robert Kilroy-Silk joined the party, and posed a threat to Farage's dominance in the media.

When he was searching for a political adviser, Farage, it is claimed, recommended Tony Bennett – not, to be clear, the crooner, but a researcher who had already been censured for circulating a pamphlet in which he suggested the prophet Muhammad was a paedophile. "Farage set Kilroy-Silk up to fail because he had become the darling of the party," said Sinclaire. Another member of the campaign remembers Farage's plaintive complaints that he would no longer be invited on Question Time.

Farage says that he can take no big decisions without the support of the elected National Executive Committee. Indeed, he adds, he is "often frustrated at the lack of real power I have within Ukip". And in a later email he scoffs at the idea that he was jealous of Kilroy-Silk, saying that he was the one who got him to join in the first place. "It is very nice of people to compare me with Machiavelli," he writes, "but it > is simply not true. I do not want to do media seven days a week and would welcome other high-profile figures."

All of this comes against a background of endless recrimination: few of the people who understand Ukip come without an agenda. Although it refers to events of some years ago, a dark reference in Flying Free to "a splinter group under the influence of Rowan Atkinson's brother Rodney" seemed to summarise the party's intense, if miniature, rivalries.

Most of those who speak against Farage today could be seen as having an axe to grind: Nikki Sinclaire, for example, was expelled after she refused to sit with European allies who she viewed as extremists. (She has also been arrested over expenses claims; and, to further complicate things, went on to win a sex discrimination case against the party, largely because whoever was supposed to file their defence missed the deadline for doing so.) MacKinlay and Campbell Bannerman also had fallings out of one sort or another. As well as members of the rank and file, Farage has rowed publicly with four of his six predecessors as party leader.

Farage, for his part, yearns for the simplicity of his previous life. "In business," he harrumphs, "when you sack people or you get sacked, you're cheesed off. And then a year goes by, and you meet them in the street, and you say come on, you silly old bugger, let's go for a drink. You move on. Politics is different, because there's no way back. So you pay your membership fee and you join the bitter-and-twisted club."

Is he difficult to work with? "I genuinely don't think I am. I would say this to you: I'm one of the most trusting people you'd ever meet. I trust everybody until they let me down. And then I never speak to them again."

Since Ukip's troubled conference, as Godfrey Bloom has chuntered on, accusing Farage of being out of touch, his erstwhile leader has tried to look ahead. Work is afoot on plans for a stunt, in the event that he is excluded from the debates that will precede the next general election, as Sky News seems determined to ensure that he will be; there are, meanwhile, rumours that a seat has been selected, South Thanet, for a challenge to Europhile Tory incumbent, Laura Sandys.

It should be a thrilling time for Farage. So why does he seem a little... tired? Partly, it's because he works so hard, regularly working weekends and putting in shifts worthy of a cabinet minister. But, he says, it may also have to do with the third time he nearly died: the plane crash on the eve of the last general election. The crash itself was bad enough; more frightening still was the moment afterwards when he thought that the plane's engine was about to explode, and burn him to death before he could get out.

If his previous scares somehow made him more energetic, this one took it out of him. "Healthwise, this job is absolutely bonkers," he says matter-of-factly. "I may have to rethink it a bit. Maybe I'm going to have to box a bit more clever as time goes on. I've been a bit smashed up, you know. I used to be like the Duracell bunny. I used to work all day, drink all evening, not bother with sleep at all and then do it all again. But I'm not as strong as I used to be."

Politicians are not supposed to admit this sort of thing, and it is hugely to Farage's advantage that he is willing to do so, particularly when it so goes against his personal brand of Tiggerish enthusiasm. There's something else curious about it, too: in a funny way, being a little tired might make him better at his job, a more convincing, sober politician for the real world than he has been able to show himself before. Knackered though he is, he still believes he can take his party forwards. The question is: do they want to go with him?

A few Farage facts

Peak golf handicap: 4

Estimated sum received in expenses over a decade as MEP: £2m

Hours allegedly spent in Strasbourg strip club on 'inadvertent' visit: 3