Hillary Clinton was going to be president. She knew it. So did her chief strategist, Mark Penn, the man who had once delivered victory for her husband Bill. Barack Obama was "unelectable", Penn declared early in her campaign, "except perhaps against Attila the Hun". How wrong can you be?

As Hillary accepts the consolation prize of a place in Obama's cabinet, the man widely believed to have blown her chances with a disastrous election strategy has come to London. Penn is speaking at a conference for business leaders, sharing his theory about "microtrends" – relatively small groups of people who can have enormous influence on politics or business, such as the "soccer moms" targeted by Bill Clinton and Penn in 1996 to great effect. But first he has agreed to face The IoS.

Let's start with a tough question: low sofa or hard-backed chair? Anyone insecure would take the chair, trading comfort for the chance to look down on the interviewer. Not him. "I'll take the sofa," says Penn quickly, almost falling into its white leather then spreading himself wide.

Download the new Independent Premium app Sharing the full story, not just the headlines

He is – and I have to be careful here, people in glass houses and all that – fat. A little boy's Just William haircut sits oddly on top of a big, shiny face. His size is worth mentioning, because it seems the fleshy embodiment of the daring, the chutzpah, the enormous self-confidence that have seen him work as a pollster and strategist for names including Clinton and Blair and steered companies such as Microsoft through controversy. "The Master of the Message", as Time called him, is chief executive of Burson-Marsteller, a global PR company. As such, I don't think this 54-year-old is used to being asked direct questions about himself. So here goes.

Mr Penn, you blew it, didn't you? Were you so interested in microtrends that you completely missed the huge desire for change? "Well, no," he says, taken aback. "I think that, you know, I think ... the book is really a non-political book."

Yes, but we're not talking about the book. We're talking about the view in Washington that he threw it all away. The accusations of arrogance and complacency. He was the one who told Hillary to play tough, model herself on Margaret Thatcher and refuse to apologise for supporting the war in Iraq. When others urged her to show a little more humanity, Penn reportedly said: "Being human is over-rated."

This is a man whose friend once described him as having "the IQ of Bill Gates and the emotional intelligence of an eggplant". Now Penn says, in his surprisingly high-pitched voice: "Hillary ran on a theme of ready for change, ready to lead. Change was always a central and important part of her message." That's strange. I could have sworn it was more like, "Don't trust the new guy, vote for someone with experience". But anyway. "Look, at the end of the day, they both got about 18 million votes. It was the hardest-fought primary in the history of America. And it turned out that some small groups switched and became critical players." Which is a pitch for his theory, obviously. "At the end of the day, they are separated by only 85 delegates out of close to 4,000."

None of which answers the question. Let's try again, being more specific about one of his biggest mistakes. People wanted a break with the past, Hillary could not provide that, and emphasising her experience only made it all the more obvious, surely? "The only thing I can tell you is that they [the Obama people] ran an excellent campaign," he says. "We regarded him as a strong challenger throughout the entire period. There was no question that he had the personal abilities, the resources, all of the things necessary to win."

Really? So why did Penn say in a memo of March 2007 that Obama was unelectable? "Huh. No. It doesn't say that at all." Yes it does, if the facsimile published by Atlantic Monthly magazine is correct. The great communicator appears thrown. "Those memos, right, that came out, were really ... er, were really, I think, show you, you know, just a piece, because ... a small part, a piece of how we were looking to, I think, set up or solve the fact that he was a very strong candidate."

Right. Now, you could say it is unfair to run every pause and splutter in a sentence like that, but it does give a sense of Mark Penn floundering. There is sweat on his top lip. The same memo shows how spectacularly he failed to see that Obama was in tune with the times: "All of these articles about his boyhood in Indonesia and his life in Hawaii are geared towards showing his background is diverse, multicultural and putting that in a new light. Save it for 2050."

Daily coronavirus briefing No hype, just the advice and analysis you need Enter your email address Continue Continue Please enter an email address Email address is invalid Fill out this field Email address is invalid Email already exists. Log in to update your newsletter preferences Register with your social account or click here to log in I would like to receive morning headlines Monday - Friday plus breaking news alerts by email Update newsletter preferences

Plenty of people were as dismissive, but none of them was supposed to be the world's best political soothsayer. Worse, Penn made an accusation that may haunt him, about Obama's background. "It also exposes a strong weakness for him – his roots to basic American values and culture are at best limited. I cannot imagine America electing a president during a time of war who is not at his centre fundamentally American in his thinking and in his values."

Not fundamentally American? That's a very serious accusation to make in US politics. Even Sarah Palin wasn't that direct. "Well, really," says Penn awkwardly, "it really just says [Hillary] should emphasise her roots ... I think that at the time he was touting a more internationalist outlook."

Penn urged Hillary to "explicitly own 'American'" in her campaign, in a way Palin and John McCain subsequently tried to do. The former chief strategist insists his advice wasn't bad, it just wasn't followed far enough. "We really didn't run an aggressive campaign until we ran the 3am ad."

That television advertisement in March showed children sleeping in their beds as the presidential emergency telephone rang at 3am. Who would viewers prefer to pick it up, the novice or the woman who knew about life in the White House? "It was very successful." It was also scary, nasty and a bit misleading (Hillary had never faced a call like that). Gloves off, it went against the national mood. Some say that kind of misjudgment is what did for Hillary. Penn says: "I think perhaps the key would have been being that aggressive early on. But, you know ... that's history."

So are his dreams of influence in the White House, even if Hillary is now secretary of state-elect. "I really couldn't be more pleased about how this has worked out," he insists. But Penn is unlikely to be anywhere near her team. He resigned from her campaign in April, after it emerged that he had simultaneously been advising the government of Colombia on its push for a trade deal opposed by Hillary. She is said to have felt let down.

So now that Obama is about to be president, does he regret saying nasty things? "At the end of the day, having a few sentences or ramblings going back and forth in the heat of campaigns, I don't think that's to be mistaken for the genuine respect I had and have for him and that Hillary did."

There's more. "It just may have been his time. He was a phenomenon. He is putting together a stellar cabinet. We have big problems; he's putting together a big government."

Can drawing strong opponents close, as Abraham Lincoln did, really work? "I think teams of rivals are tricky things," Penn admits. "Obviously, in this case I am really enthusiastic about it." Obviously. "She went out working very hard for him in the election campaign. I think that made a big impression."

The Washington Post says Hillary still owes more than $5m (£3.4m) to a Penn company. "There is a debt remaining and she has consistently said she is going to clear that up."

Mark Penn is probably not desperate for the money. His house in Georgetown, Washington, is worth about $5m. He lives there with his second wife, Nancy Jacobson Penn, a Democratic fundraiser, and their five-year-old daughter (Penn also has three other children by his first wife). Nor does he let things get him down – rejected by Harvard as a teenager, Penn took a train to Boston and asked to see the dean in person. He got a place.

His indefatigable side could come from his father, who migrated from Lithuania to New York and ran a kosher poultry plant, or his mother, who was widowed early and worked all the hours she could to send her three boys to an exclusive school. There Penn ran a paper and conducted his first poll, asking pupils about racism. While at Columbia Law School he started a company with his friend Doug Schoen, and their research helped Ed Koch to get elected Mayor of New York in 1977. Penn built their first computer, from a kit. He and Schoen refined political polling techniques, then made lots of money by applying them to the commercial market, allowing companies such as AT&T to target customers.

Penn worked for the maverick Ross Perot in 1992 but switched to Clinton four years later. That was his greatest political triumph, unless you count helping the beleaguered Tony Blair to win a historic third term for Labour in 2005. Earlier this year there were reports of Penn being called for advice by Gordon Brown. So, for the record, has he ever been consulted by the Prime Minister? "No."

You may have noticed that Penn has so far failed to admit any blame whatsoever for Hillary Clinton's defeat. But when I ask again whether missing the big picture undermines his message on microtrends, he surprises me with an admission. "Who can possibly escape a period of flak? None of my clients ever did. So. The truth is I take my share of the responsibility. We didn't win."

He just can't help spinning it, though. "I've won a lot of other campaigns and I hope to win a lot more in the future, but I take it on the chin for [this one]. It happens to me, it happens to everybody."

So now Penn flies around the world for Burson, helping companies that "run into snags". But the one place he is not likely to appear is in the Oval Office. Not now. "Huh. Look, I've worked for well over 20 heads of state," says the man who once seemed so clearly destined to be at the side of the next President Clinton. So what if he blew it? He's big enough and ugly enough (and rich enough) to take it. "You win some, you lose some."

Penn portrait

1954 Born in New York to a Lithuanian immigrant father who runs a kosher poultry plant.

1964 His father dies. Mother works all hours as a supply teacher to send him to exclusive Horace Mann School. Goes on to Harvard.

1975 Starts a polling company while still at college.

1977 Penn and Doug Schoen help Ed Koch become Mayor of New York.

1996 Wins Bill Clinton a second term. US Pollster of the Year.

1998 Adds Microsoft to his clients, who include both corporate giants and national leaders.

2000 Helps Hillary Clinton become Senator then build a presidential campaign. Pollster of the Year again.

April 2008 Resigns from Hillary's team after it emerges he has advised Colombia on how to win a trade deal that she opposes.

December 2008 Speaks at Leaders in London, a conference for CEOs, alongside chess champion Garry Kasparov and former New York mayor Rudy Giuliani.