Douglas Carswell is having his picture taken in committee room 11 at the House of Commons. He looks stiff and unrelaxed, clenching his hands together in a poor impression of a statesman. “Ah, if only there was a painting of Gladstone behind me,” says the former Conservative MP for Clacton and newly elected Ukip MP for Clacton.

And he’s off. Within seconds, he’s lecturing me in his clipped manner on Liberal leader Gladstone (a hero), the evils of empire and vested interests, the end to true free trade and the wrongness of Tory Disraeli. I’m confused. If anything, I was expecting lectures on the evils of immigration and the EU.

Carswell is tall and gangly with a square-jawed Popeye face and sloppy combover. The journalist Charles Moore, an admirer, once described him as the kind of crank who stalks the streets with plastic bags full of documents and said that “many of his colleagues regard him as a lunatic”. Carswell is certainly complex.

He strides through the central lobby but is too diffident to hold my tape recorder. “You’d better hold that otherwise I’ll feel spooked. It makes me all self-conscious.” Self-conscious? This is the man who pulled off the year’s most audacious political stunt – and kept it secret. In August, he announced he was defecting from the Conservatives to Ukip. Then came the double whammy – he would not merely cross the floor, he would stand down as MP to force a humiliating byelection for the Tories, which he duly contested and won with a 12,404 majority to become Ukip’s first elected MP. The shy, retiring Carswell is the same man who bowed showily to William Hague and Michael Gove on re-entering the Commons before giving a sly nod to Tory rightwinger Peter Bone, also rumoured to be on Ukip’s hitlist.

As we head off to the cafe, he stops briefly for another impromptu lecture when he sees signs for a lobbying meeting. He peers at the names disapprovingly. “Who d’you reckon funds them? That’s the key to transparency. I was on the Armed Forces Parliamentary Scheme and got thrown off it for asking questions about funding and corporate governance.” He buzzes with manic energy.

We pick up a coffee, and sit down. He doesn’t normally breakfast here, but he has no office. “It should be sorted out by the end of the week,” he says.

In some ways, Carswell is classic Ukip material – rampant libertarian, anti-European, anti-gay marriage (on traditional Christian grounds), anti-human rights lawyers, a ferocious basher of legislation protecting workers’ rights. But in other ways he is surprising – a moderniser, technophile, and obsessed with transparency and direct democracy (Ukip’s MEPs famously voted against transparency on expenses).

His parents were doctors, and he grew up in Uganda. His Scottish father was employed by Idi Amin’s government (the inspiration for Dr Nicholas Garrigan in Giles Foden’s novel The Last King of Scotland) and diagnosed the first Aids cases in Uganda. Carswell, 43, is proud of attending racially mixed schools, and it soon becomes apparent he is unlikely to bang on about banning foreigners with Aids from entering the country, rail against the number of foreign accents on public transport or worry about the prospect of a Romanian neighbour as his Ukip leader Nigel Farage has done.

Carswell decided to quit the Conservatives earlier this year, and had a traumatic time of it. “I was very stressed in the six months before I announced it. July and August were horrible. I knew I had to do this, and I was terrified. I had difficulty sleeping.”

Once he became disillusioned with the Tories, he says, he had three options – quit politics, fight to change the party from within, or oppose it. What stopped him quitting? “I thought no, what am I going to tell my five-year-old in 20 years’ time when she asks: ‘Why did you stop being an MP?’ I’ve got as much right to be in politics as David Cameron.”

Nigel Farage and Douglas Carswell on the day of the Clacton byelection. Photograph: David Levene/Guardian

Ah, Cameron. Carswell says he hates personalising politics, but you sense something profoundly personal in his fallout with the Conservatives. One of Carswell’s bugbears is an overly powerful civil service. In 2012, at prime minister’s questions (PMQs) he said: “A few weeks ago in this House, I asked to what extent the prime minister believed the Sir Humphrey machine was frustrating reform. He said it wasn’t. Last week in Asia, he said Yes Minister was true to life. Can he say what has changed his mind?” Cameron’s reply was brutal: “I think the Hon Gentleman does need a sense of humour.” They didn’t have much of a relationship by then (Carswell voted against the whip more than 60 times) and had even less of one afterwards.

Changing the party from within was impossible, he says. “That’s what half of the good guys in the Tory party do and they end up grumpy and disappointed. In 20 years’ time, I don’t want to be a Tory MP who hides their disappointments by saying: ‘Look, I’ve been given this bauble by the whips, and I had a minor influence on this’.”

Which left option three: “Put my dilemma on the table; ask the voters to resolve it with me.” He looks at me, elated, with huge, staring eyes. “I thought I can lose in an electoral sense, but I’m still going to win. Do you know why? Because I’m never, ever, ever, ever going to grow up into a bitter Tory backbench MP.”

Carswell’s defection is still something of a mystery. He recently said the only way to guarantee a referendum on Europe is to ensure the Conservatives win the next election. So why walk now? “Because Cameron’s not serious. It’s a smoke-and-mirrors referendum. His advisers told me the plan; it’s to work out from focus groups and pollsters what it would take to get the soft ‘outers’ and the undecideds to stay in, to offer them that, and once that hurdle is cleared to stick with the status quo.”

But, he says, to really understand why he fell out with the Tories, you have to go back a decade. “I started working in the policy unit referring to one D Cameron to put together ideas for the 2005 manifesto. Up until then, the Tories in opposition were torn between reheated Thatcherism or patrician Toryism. They had no compelling critique of the state of Britain today, what they wanted to change. They had no intellectual coherence.” Carswell wanted to make the party and politics in general more accountable. Some of his ideas made it into the 2005 manifesto, such as directly elected police chiefs. After that, he spent another five years refining ideas about direct democracy, such as recall (whereby MPs could be voted out by constituents if more than 50% decided they were not up to the job) and open primaries (to ensure candidates aren’t parachuted into safe seats). Carswell’s proposals contributed significantly to the 2010 Conservative party manifesto. “I came up with a whole bunch of wonky, boring but really important stuff about how parliament will hold ministers to account. And I voted for Cameron because I thought that he got this.”

But Carswell discovered that he didn’t get it. “In February, I brought out a copy of the manifesto sitting on my shelf and reread it and it came as quite a shock to me because of the gap between what we’d promised and what we’d done. The failure to deliver on political reform is what has driven me.”

Oh, come on, I say, we all know what politics is like; you can’t really be shocked. He looks at me, intensely serious. “It’s news to me. I didn’t think it was like that. Maybe I’m learning. But it shouldn’t be like that. And if you had recall and open primary it wouldn’t be like that. Why do we have to be governed by these mediocrities? Why do we have to be governed by people who read what they read in the Economist and the FT and regurgitate all the failed cliches that got us in to this mess? I think they’ve just watched Richard Curtis’s Love Actually, and think it’s a manual for how to govern the country.”

What does he mean? “They want to hold office, but they don’t know what to do with it. They watch The West Wing. They think it’s all about giving speeches, positioning, going to Washington. They’re not in it for change, they’re in it to hold office.”

He throws a fact at me to prove his point. “Seven out of 10 seats are safe seats so career politicians get parachuted into safe seats and they only answer to career politicians. They’re out of touch. You get MPs who will honestly tell you they’ve never voted against their party on a three-line whip.” He looks appalled. “How can you possibly do that?”

Did he ever vote with the party? He smiles. “Of course, of course. I wasn’t one of the most rebellious MPs by a long chalk.” Who was? “I think it’s the great Philip Hollobone.”

Carswell celebrates his byelection win. Photograph: Peter Macdiarmid/Getty Images Europe

As we talk, Dennis Skinnner strolls in to the cafe, and sits by himself nursing a cuppa. “I’ve always got on with Dennis. He’s got a great sense of humour. He’s a nice man.” He says he wants to get in early to PMQs to secure his seat by Skinner on the alternative opposition frontbench. Is he putting down a marker? “No, I’m not confrontational, that’s not my style. I’m almost doing it apologetically.” So why is it important to sit there? “It’s a few seats away from government ministers. My Clacton mandate is as good as anyone else’s.”

The cafe is a dissident’s paradise. A few minutes later, Peter Bone walks in, greets Carswell enthusiastically and sits directly behind us. He’s closely followed by Labour rebel Graham Allen.

“Hi, Graham, keeping well, comrade?” Carswell says. “Fantastic guy,” he says, once Allen is out of hearing.

Some political commentators have suggested that Carswell is almost Bennite in his approach to direct democracy. He seems delighted when I mention it. “Benn said the key questions were: who has power, who gave it to them, on whose behalf do they wield it, and are they accountable? I remember thinking this guy is spot on.”

The more time I spend with him, the more bewildered I am. Why on earth did he join Ukip? Well, he says, he could hardly have moved over to Clegg’s lot. “If the Liberal Democrats hadn’t made 20 years of errors, yes. But they have made it very clear that they’re not liberal and that they’re on the side of the elites, not the people.” If Gladstone was such a hero, why didn’t he stand as an independent one-man Liberal party candidate? He laughs. “Well I toyed with the idea. But I want change for the country. And to change the country you need 300 and something MPs, so political parties are necessary.”

Carswell was never going to be one of Cameron’s cronies. He might have gone to the public school Charterhouse when he returned to England, but he didn’t end up at Oxbridge. He says he had a great time at the University of East Anglia studying history, but of course he’s aware that it separated him from the insiders. He talks bitterly of “all the people who went to the same colleges, and read PPE, which means they all think they know everything about everything … and we wonder why the country is so badly run.” But for all his bravado about being an outsider, he does seem incredibly hurt that he wasn’t listened to; that he never was one of them.

Does he want to see Cameron beaten? He doesn’t answer directly. “The balance of power for Ukip would be good.” Does the Tory party need a new leader? “It needs a lot more than a new leader.” He believes the Tories have become outmoded because they still act as if there has been no digital revolution; as if you can only communicate top-down via mass media. “The internet turns those assumptions on their head. You can re-personalise politics. The moment I got a blog and a Twitter account I started thinking I can communicate directly with people. I don’t need to be a pale imitation of Harry Enfield’s Tory Boy.”

Carswell has enjoyed his return to the Commons. What has Cameron said to him? “I’ve not spoken to him. Michael Gove has been very nice, very helpful. He phoned me shortly after I won. I trust him. If Michael Gove gives you his word, you can hold him to it, and there are not many people on the frontbench I would say that of.”

I ask whether he’s grooming future defectors and he looks appalled. “If I swanked around this place smugly suggesting to people they should cross over, it would be obnoxious, counter-productive and it’s not who I am.”

So much of what Carswell says makes sense. But there seems to be a disconnection in his thinking. I still haven’t really got the foggiest why he’s joined Ukip. It feels like there’s an existential crisis, or psychic trauma, at the heart of it, rather than any affinity with his new party. For somebody who professes to be so optimistic about the future, why move to a party shrouded in pessimism and the past? “Let me explain why. You’re partly right, but mostly wrong. There was a public meeting in Jaywick in my constituency. Biggest public meeting I’ve ever spoken at. People asked questions that betrayed a legitimate anger with politics and a legitimate pessimism about things. I answered those questions by trying to show that things can be better, and the response was extraordinary. The biggest cheer I got was when I said: ‘Can’t you remember the Olympic Games two years ago, can’t you remember how good that made you feel about Britain as it is; that image of our country all together, all as one, didn’t that make you feel proud, didn’t you all feel part of Team GB when Mo Farah won that medal?’ And people said: ‘Yesssss!’”

But you’ve just joined the party that fielded a candidate who asked how Mo Farah could win gold for Great Britain when he is “an African from Somalia”.

“Who’s that?” Carswell stammers.

David Wycherley, I say.

“Well, that’s clearly wrong. Is he a candidate now? If he was a candidate now, I think there would be issues with that.”

Wycherley subsequently stood in the May local elections and is still a party representative for Walsall.

Carswell has been assured by Farage that there is no room for intolerance. “Simon, one small favour. You’re right to scrutinise Ukip and point out those comments are wrong and indefensible, but could you please apply the same scrutiny to the Conservatives? It was a Conservative MP who tweeted obnoxious tweets attacking the Olympic games opening ceremony.”

As we head off for PMQs, I tell him I can’t imagine him getting on with Farage – he doesn’t even like drinking, for God’s sake. Look, he says, it’s about time Farage was given credit. “We have to acknowledge the extraordinary tenacity of the man, and the personability of the man.”

I know all about his personability, I say, I’ve shared a breakfast pint with him. “You’ve put your finger on it,” he says enthusiastically. “There are stylistic differences. I don’t like booze, but I love McFlurrys. I try to get the guy to go to McDonald’s for ice-cream after we’ve been out knocking on doors.”

Carswell is on a high, bouncing down the corridors of power like Zebedee. “When I came back into this place, everything seemed so familiar but, I kid you not, the corridors seemed better lit – then I realised that was me. That was the happiness, the serotonin levels in my neural networks.”

He mentions a picture he had on the walls of his campaigning office. “I’ve kept it as a souvenir. We’ve got a picture of Gandhi and his famous quote about what you do and what you say and what you think being in harmony. People say why do you have that up there? Well maybe the reason there is so much political unhappiness in the country is because, for people doing politics, what they say and what they do and what they think are three different things.”

The following day he rings back to answer a few queries. I tell him I’ve been thinking about all he’s said, and wondering how hurt he was when Cameron said he needed to get a sense of humour. Yes, he says, he does think it was unkind. He pauses. “Ah well, I’m definitely not having the last laugh now, am I?”