"Helloooh!" shouts a slightly undressed man from the top window. "Down in a second." Paddy Ashdown pounds down the stairs and opens the door, a little breathless. No socks, no shoes, shirt not quite buttoned, belly bursting from unbelted trousers. He apologises, and explains he wasn't expecting me so early. "Right," he says, leading me to the kitchen of his London home, where a pack of fags and a packet of salmon lie on the table. "Would you like some salmon?"

"Paddy, it's 9.15am. Bit early," I say.

"Not if you didn't have dinner last night!" he guffaws, picking at the pieces.

Ashdown is a youthful 72, and has packed so much into those years – marine officer, spy, failed heavy-goods-vehicle driver, youth worker, leader of the Lib Dems, lord, governor of Bosnia, author. His new book, A Brilliant Little Operation, is subtitled: The Cockleshell Heroes and the Most Courageous Raid of WW2. I ask what attracted him to the story of the Special Boat Service mission to blow up the German merchant fleet anchored at Bordeaux harbour in December 1942.

"Come on!" he says incredulously. "What d'you think?" He has a point.

The story of Blondie Hasler and his 11 fellow raiders couldn't be more apt for Ashdown to tell. Twelve men set off in six tiny canoes to destroy the huge ships. The project was as stupid as it was audacious; as tragic as it was triumphant. Yes, Blondie and a few of his men eventually paddled to their targets and planted limpet mines on the ships, but it turned out the ships were empty (the whole point was to stop them transporting cargo), the men were betrayed by a senseless secrecy at the highest level, and nearly all of them died; drowned or executed. And yet, pyrrhic victory though it appears, the mission demoralised the Germans, pricking their bubble of invincibility.

Ashdown himself was an SBS officer, and the more you read of this moving story, the more you see how closely he relates to Blondie, the leader of the Cockleshell Heroes. In 1965, as a young man who had just qualified as a swimmer canoeist in the Royal Marines SBS, he met Hasler on a train but didn't recognise him. In the book, Ashdown describes how Hasler quizzed him, but he gave him short shrift. "'Look, I'm not at liberty to tell you the details of my job. It's confidential and not to be discussed with those not entitled to know,' I replied in a pompous and irritable tone."

When he was told it was Hasler he had been so rude to, he felt ashamed. And that shame has stayed with him ever since. After all, he says, Hasler's mission was a defining moment in SBS history. "It was probably the event that showed Special Forces were not just going on shore, slitting a couple of German sentries' throats and coming out again."

Ah, throat-slitting. Classic Paddy territory. Ashdown is a little bit in love with his own mythology as the Lib Dems' very own James Bond. At the same time, he's rather coy, or discreet, about his previous – while he's happy to hint at dark and dangerous undertakings, he suggests that it would be improper to go into details. The inside sleeve to A Brilliant Little Operation describes him as "The only Mandarin-speaking trained killer to rise to prominence in British politics".

"So by the time you joined the SBS its throat-slitting days were over," I say.

"Hahaha! Hahahahaha!" He laughs loud and theatrically. "You are so predictable, you guys! ... Have you ever slit a throat? So predictable!"

"Well, you always avoid the answer," I say.

"Yeah, you're right I avoid the answer. I'm not answering that question, so forget it."

Ashdown with John Major and Tony Blair at the VE Day celebrations, 1995. Photograph: Rex Features

Actually, he is far more interesting when he dampens down Action Man Ashdown. "I was trying to make the point about Hasler that he never slit anybody's throat. He never killed anybody. He didn't like violence, and I didn't either. After a little war I fought in Borneo – God knows it was nothing in comparision to what these guys did – I wanted to conduct my soldiering more through guile than through violence. So it's actually not about slitting throats, that's the whole point of it." Ashdown points out that the SBS's motto used to be "Not by strength, but by guile" and regrets it has become: "By strength and by guile."

The previous day I had mistakenly turned up at his house, 24 hours early for the interview. Ashdown wasn't at home, but his wife Jane was. She's a lovely, warm woman, who speaks the language of the barrack room. I suggest that Paddy and Blondie had plenty in common. "Oh no, he wouldn't assume ... Blondie was an absolute hero," she said. Then she thought about it. "Awkward bastards," she said. "Both awkward bastards."

I mention it to Ashdown and he throws his head back and laughs so much I think he's going to choke. "Me, an awkward bastard? Listen, I'm a quiet little herbivore. Aren't all Liberals nice, cuddly, herbivorous creatures on the border of politics? Hahahaha! You're such a shit; you go and interview my wife and use it against me."

But he concedes that they did have plenty in common. "He was an outsider, I was too. He was regarded by his friends as a bit unclubbable, and always testing the limits of his endurance, and I was. And the extraordinary thing is that this chap went through the same thing; he voted Liberal all his life and was always trying to find the alternative way of life."

Actually, in his marine days Ashdown was a socialist. "Not a very popular thing to be as a serving officer in the 1950s. When I commanded my first troop I discovered there were people who couldn't read and write, and I was scandalised. I thought: how could you have a nation that does that? I was a Labour party supporter till 1969."

So what changed him? "The thing that made me a liberal was in the SBS I was commanding men who were better men than I was. And I was in the position of being an officer because I had had a privileged background and had gone to a public school."

In what way were they better? "Better soldiers, braver, tougher. I would have relied on them anywhere. So the revelation to me serving in the SBS was that if you could only have a country that was genuinely meritocratic, in which every person was judged by their ability, that would be the kind of country I would want to be part of. So here I believe is the difference between a socialist and a liberal. A socialist believes everybody is equal in terms of outcome, or should be. I believe everybody deserves to be treated as equal in terms of their status as a human being, but the outcomes they produce are totally different."

After leaving Labour in 1969 he found himself in the political wilderness – sceptical of all party politics. And then he heard a rat-a-tat-tat on the door. "This is a bizarre story. It was 1974 and I'm off out to Geneva to the Foreign Office … or some strange part of it." Ashdown likes to allude to his time as a spy without saying the word. Don't you mean MI6, I say.

He affects a disapproving tone. "Well, you can say what you like about it. I was a diplomat in Geneva. Anyway, I was on my way out to this posting, and a more unlikely angel of conversion you could not imagine. A bloke knocks on my door and I can't remember if he had sandals on, but he certainly had a cagoule on." He impersonates an insipid Liberal stereotype. "'Hello, I'm from the local Liberals.' I said go away, and he said: 'Well, I'm canvassing for the election.' So I said: 'OK, come in, talk to me and if you can persuade me to be a Liberal I'll vote for you.' From that moment on I took liberalism down from the peg that it seemed to have been hanging on as an overcoat all those previous years and it's felt totally comfortable ever since."

He left Geneva in 1976 to stand for the Liberals in his home constituency of Yeovil. It seemed a hopeless task. "The Tories had been in power in Yeovil since 1910, and my leader [Jeremy Thorpe] was then being arraigned for conspiracy to murder at the Old Bailey."

It took him eight years to win Yeovil, during which time he was unemployed for 18 months. "We were down to our last 100 quid, and I said to Jane: 'What should we do?' And we said: 'Let's send the kids for a holiday to some friends in Switzerland.'"

Did he wonder why he'd given up the well-paid job in Geneva? "Of course, yes. It was a mad decision." And it was a job he had enjoyed? "Loved it. There's no job I've had that I haven't enjoyed."

It was during this bleak time that he applied for a heavy-goods-vehicle driver's licence. "I went off to do the aptitude test and the bloke said: 'Do me a favour, never drive a heavy lorry; you're absolutely lethal on the road.' But there wasn't anything I didn't try." Perhaps it is this struggle that has shaped him even more than the clandestine glitz of the marines and MI6 – and explains his ability to communicate to all sorts. In A Brilliant Little Operation, Ashdown writes that Blondie Hasler had no money for much of his life. Isn't that something else they have in common? He balks at the suggestion.

"Well, that's not true. That's not true, no, no, no. I've always been desperate about money. Not to get lots, but to have enough."

He says his fear of poverty goes back to his father, a former officer in the Indian army who returned to Northern Ireland to run a pig farm. "Remember, I'm the oldest of seven children, my father's business went bust in Northern Ireland, he emigrated to Australia with my six brothers and sisters." He stops to correct himself. "Five brothers and sisters because one had died, and I was left behind. I watched the man I admired most in the world broken by the fact that he didn't have money. I'm obsessive about making sure my family has enough money. Blondie didn't care about money, he didn't bother to make money, but for me making sure my family has security matters a lot."

In 1988, only five years after winning his seat, Ashdown was elected leader of the newly merged Liberal Democrat party. He was just what the party needed – tough, direct, a natural headboy. Does he think he could have been prime minister as leader of the Lib Dems? "I'm pretty pissed off I haven't been." Again, he throws his head back and roars.

"But you mean it, don't you," I say.

"It spoils my whole afternoon that I haven't been prime minister. Hahahahahhaha!"

With wife Jane at the time of the Paddy Pantsdown scandal, 1992. Photograph: Michael Stephens

"But you absolutely mean it," I repeat.

"Look, nobody wants to go into politics and do it at a high level without wanting to do it at the highest level. If you are leader of a party, what d'you want to do? You want to be prime minister. Course you do."

In 1992, he admitted to a past affair with his secretary and the Sun reported it under the headline: It's Paddy Pantsdown. Did that put an end to your ambitions to be PM? He bristles. "No." It didn't make any difference? "No, it improved my poll rating." Really? "Yes, of course it did." He smiles. "By the way, don't imagine I recommend it as a way of improving your poll rating."

So Paddy Pantsdown was good for you? And now he is wholly serious – there isn't a hint of bluster or braggadocio, just a very raw vulnerability. "It was terrible for me. Terrible for me. You have no idea. It was terrible for almost every aspect of my life. I thought the party was seriously damaged and it wasn't, but that was a tiny glimmer of light in what was an irretrievably bad period."

Was he a victim of a fantastic headline? He shakes his head. "Well, if you put yourself in a position where you can be done that kind of damage by a Sun headline, you're a fool. Aren't you? So don't blame the Sun, blame me."

It would have done for many politicians, but Ashdown rode the storm, and remained leader for another seven years. He became high representative for Bosnia and Herzegovina in 2002, and was widely praised for his healing stint in the brutalised country. After that there was another shift in Afghanistan at the request of Gordon Brown. Ashdown says he didn't want to go, but when you are asked to perform a public service, what do you do?

A few weeks ago he launched a leonine defence of the Lib Dems, and more specifically leader Nick Clegg. Today, he does the same. He regards those who feel betrayed by the party as weak or naive – notably Guardian leader writers who backed them in 2010. "The Guardian feels like a jilted lover. It hates the Liberal Democrats. The Guardian feels personally betrayed because for the very first time it gave the Liberal Democrats its support and what did we do? We went off with the Tories. But what else would you have done in the circumstances?"

What about the first-time voters who voted Lib Dem because of their promises on student tuition fees, and because they believed they were a radical alternative to the main parties? Again he says it comes back to the single question, which he spells out in one-word sentences. "What. Would. You. Have. Done?"

Well, I say, many voters might say it was right for the Lib Dems to team up with the Tories but it was wrong for them to vote Lib Dem in the first place and won't do so again. "But what is the point of being in politics if you are not able to influence the government of your country? And it is the eternal battle for any thinking politician – between principle on the one hand and what is practical in effecting the government of your time. I think at the next election the Liberal Democrats will get a much bigger dividend than most commentators think because we'll have seen this thing through and stuck with it."

What if the polls prove correct, and the party does lose half its vote? "Even if the party were terribly damaged at the next election, I would still think it was the right thing to do."

Ashdown knows all about conflict resolution – there was his four years in Bosnia for starters. "Sarajevo is one of the greatest cities in the world. I speak Bosnian, and there's a great Bosnian saying: 'It's easy to beat thorn bushes with other people's pricks.' I said to Nick it will make an excellent motto for the coalition."

But for now Baron Ashdown of Norton-sub-Hamdon insists he has other things to concentrate on – seeing his two children, both teachers, whom he calls his best friends; racing his grandchildren on the ski slopes; and writing more books. Which all sounds perfectly feasible. But there were rumours that when Liam Fox resigned as defence secretary he was hoping for a call. His face scrunches into a grin.

"Was I saying: 'My God, I hope a call comes! I cant wait! I'm so disappointed!'? No! Can I repeat the phrase that I have not the slightest hankering of going back into government." But if the coalition did come knocking? "I'm very happy as I am." He stops, and silently weighs things up. "But if the government asks me to do things, I'm very happy to do them."