When you have been secretary general of the United Nations for 10 years, it's never going to be easy to slip into humdrum anonymity. Kofi Annan gave it a good go. On leaving office he and his wife borrowed a friend's Italian hideaway, deep in the forests near Lake Como, and retreated into eight weeks of blissful solitude. With no TV, no radio, no papers; Annan was free, finally, from the clamour of the world's troubles. Two weeks in, he began to get bored.

"Let's go to the tobacconist in the village," he suggested, "and buy a paper."

They had been in the shop for less than five minutes before his heart sank. A group of men were gathered in the corner, staring. "Oh no!" he whispered to his wife. "We've got six weeks to go, and we've blown our cover. How are we going to manage?" One of the men approached and thrust out a pen and paper. "Morgan Freeman, may I have an autograph?" Annan flashed his best Hollywood smile, scribbled "Morgan Freeman" and fled.

"So when people say: 'Here's a man who needs no introduction,'" he chuckles. "I say to them, careful!"

As he's telling the story it occurs to me that a tourist's failure to recognise his successor would not constitute an amusing anecdote. Annan does have a curiously movie-star presence, flawlessly crisp and unusually still, but five years after leaving office he remains a more commanding presence in world affairs than Ban Ki Moon. "I thought retirement was going to be easier," he smiles, "but I discover it's hard work. I should do what Mandela says and retire from retirement one day."

We meet just a few weeks after he stood down as the UN's special envoy to Syria, and his dismay at the ongoing carnage clearly hasn't faded. It's equally clear, however, that he saw no choice but to quit. "The first thing I said to the security council was: 'This is a near-impossible task. I'm going to try, but I can only do it if I have your united and sustained support.' Because you need that support in order to put pressure on all the parties. 'United' is key. Be united." But the security council remained divided, and as a consequence, Annan says gravely: "The Syrians are going to pay the price. It's an innocent people caught in the middle."

The plight of Syria is, in a sense, an extension of Annan's new book, or an illustration of its fundamental message. Interventions: A Life In War and Peace tells the story of his 50-year UN career, and of the emerging role of UN peacekeeping. It's a role with which Annan has been intimately linked, but humanitarian intervention – like Syria's future today – lies ultimately at the whim of the security council.

Kofi Annan quits as international envoy to Syria. Reuters

Until the early 90s, UN peacekeeping efforts had been minimal, most conflicts being adjudicated by the world's two superpowers, whose hostility ensured a security council too divided to authorise more than a handful of peacekeeping missions. Between 1987 and 1992 barely a single UN mission consisted of more than 100 observers, facing little personal risk. By 1994, however, 80,000 UN troops were deployed in highly dangerous warzones. Annan took charge of peacekeeping in 1993, and his book provides a fascinating account of the new challenges facing a security council that, no longer paralysed by division, finds itself the repository of global demands for something to be done about pretty much every problem in the world.

"Time and again, when member states and the governments are faced with an insoluble problem, and they're under pressure to do something, that something usually ends up being referred to the UN." He allows a wry chuckle. "The security council is dealing it with it, so they can relax. And you get the mandates, yes. You just don't get the commensurate resources to deal with it, so you are bound to run into difficulties and fail. And then be blamed. One of my predecessors used to say the letters SG [secretary general] stood for scapegoat."

The new appetite for humanitarian intervention was in its honeymoon period in 1993, when UN forces were committed in Somalia to what then US secretary of state Madeleine Albright declared "an unprecedented enterprise aimed at nothing less than the restoration of an entire country". But after rebels downed two US helicopters, the Americans pulled out, and the UN mission collapsed. The legacy of this failure, Annan writes, explains why the UN stood by and watched a massacre unfold in Rwanda a year later. When the world belatedly woke up to the genocide, UN peacekeepers were blamed for member states' reluctance to risk their own troops' lives.

When it came to Kosovo in 1999, Russian support for Serbia ruled out any chance of a security council resolution mandating troops. For the first time in the history of the UN, its secretary general endorsed military action conducted by Nato without the authorisation of the security council. It was, by Annan's account, the ultimate moral dilemma; do loyalties lie with the UN and the rule of law, or with innocent civilians being slaughtered? He stands by his support for the Nato airstrikes, but warned at the time that unless the security council was restored as the sole source of legitimacy, the world would be "on a dangerous path to anarchy". That prediction proved horribly prescient four years later, when the security council failed to pass a second resolution authorising an invasion of Iraq, and the war went ahead anyway. The war came close to costing him his job – but to many of his admirers it became his finest moment.

Annan was always doubtful about the extent of Saddam's hidden WMDs. "The UN inspectors had been in there for quite a long time, and destroyed quite a few arms; in fact, people argued that they destroyed more weapons than the first gulf war. So the feeling was that he may have had something, but how massive, and the nature of the weapons, was unsure." So why wouldn't Saddam co-operate with the inspections? "He had been bluffing to his neighbours for so many years, telling them: 'I'm powerful, I've got these weapons, don't mess with me.' It was a psychological bluff. So he couldn't bear to admit that actually he'd lied. I think that was the explanation."

Annan's great regret was to appoint a gung-ho weapons inspector, insensitive to the delicacy of Iraqi military pride. Richard Butler, he writes, was "a colossal mistake and one of the worst appointments I ever made". Why? "In a place like Baghdad, where the people are very nationalistic, you need to find a way to work with them. You cannot go there and throw your weight around."

He still winces at the memory of all the grandstanding that led inexorably to war. Bizarrely, in the midst of all George Bush's fighting talk, the president told Annan: "It brings tears to my eyes to think of what the Iraqi people are going through under Saddam." Annan is generous enough to say: "I think it was genuine; I don't think it would be crocodile tears," but pretty blunt about Bush's other motives. "There was quite a bit of that in the air, that he was trying to finish his dad's job. And the advisers around him also wanted that. They were also thinking, we kill two birds with one stone – we get rid of a bad dictator, and get rid of the weapons." What about the third bird – Iraq's oil reserves? "Well that's what people say," Annan agrees with a chuckle. Those who say it have often been dismissed as lefty conspiracy theorists – but when I ask if Annan would say it himself, he replies without hesitation: "Yes. It was a country with resources in a region of extreme importance to the world. I'm not sure if Iraq had been as poor as Somalia that there would have been that interest in going to save the country."

Kofi Annan in Naqura, Lebanon in 2006, flanked by UN peacekeeping troops Photograph: Zohra Bensemra/AP

He is less sure about Tony Blair's motivation. "Tony, at his best, in his heyday, would have been one of the most brilliant politicians of his time. But how he got himself entangled with this … I think there was a genuine friendship with Bush, but perhaps there was greater need on Tony's side than on Bush's. I wasn't sure if Tony genuinely wanted to go to war, or was just trying to protect the special relationship, and the special relationship trumped everything. It's difficult to say, because at the beginning he appeared to be on the right side, and he was also in favour of a second resolution. I hoped that would have been the point where you tell George: 'George, you are on your own. I'm not following you on this without the council resolution.' And I wonder what George would have said." Does he think it might have stopped him? "I think it could have made a big difference."

In 2007, Blair was appointed special envoy to the Quartet, a diplomatic mission created in 2002 by Annan himself to represent the US, EU, UK and Russia in peace talks on Palestine. I found Blair's appointment baffling, and tell Annan I struggle to see how he could … "Do it?" Annan completes the sentence. "Yeah, yeah." So he shares the bewilderment? "I share some of that. I was surprised that he accepted the role. And nothing much is happening on that dossier. For the past four to five years there has been no real Middle East mediation."

Annan famously declared the Iraq war illegal, and has never regretted it. "Because that was my genuine belief." But the price he paid took even him by surprise; "The UN got attacked, and I got attacked personally. The Oil For Food situation was used to beat me up and also beat up the UN." The US made allegations of widespread corruption under the Oil For Food programme, which Annan believes were motivated entirely by a desire to punish him, for they took a sinister turn when his own son, Kojo, became implicated. Kojo had worked for a Swiss company, which in 1998 won a contract under the programme; he left the company that same year, but then came the revelation that, unbeknown to his father, it had continued to pay him for a further five years. Annan was mortified. "And furious."

It must have tested the father-son relationship to breaking point. "Yes," he agrees uneasily, "A friend of mine said: 'It's tough for you, but I also feel for your son. You come under the microscope not because of something you've done or who you are, but because of your father – and that makes life difficult for a young person.' But I said: 'There are certain things you just don't do.'" Why hadn't Kojo told his father about the ongoing payments? "Maybe, sometimes, if you think you need help from Daddy, you probably don't share all the information. I don't know. We're talking about a boy who was 22, 23 at the time. But now we are OK."

The pressure from Washington was ferocious, but he never seriously contemplated resignation. "It may have fleetingly crossed my mind. Sometimes you feel: 'Who needs this? Why should one put up with this?' But I felt it was a witch-hunt. So to walk away and give them the feeling that they have won? No."

For all its ugliness, the onslaught against Annan was in essence no more than a manifestation of the fundamental tension underscoring his whole diplomatic career – the conflict between national self-interest, or realpolitik, and the loftier ideal of international law embodied by the organisation Annan served for 50 years.

"They clash, they clash," he agrees, with a smile. "I think we've made progress, but you are right – the national interest still often trumps the broader interest. But what governments and people don't realise is that sometimes the collective interest – the international interest – is also the national interest."

Born in 1938 into a powerful Ghanaian tribe, Annan had a privileged childhood in what was the Gold Coast, coming of age as his country won independence from Britain. Awarded a scholarship to study in America, he assumed he'd return to serve his nation, but a job with the World Health Organisation in Geneva introduced him to a "worldly and engaging environment", which made him reassess his loyalties. "I began," he writes, "to realise that community for me would mean something different from what it had meant to my father's generation."

Kofi Annan makes an appearance on Sesame Street in 2001. Photograph: AP

How does someone with so little personal sense of national loyalty get through to men for whom it is everything? I've always wondered what Annan says to world leaders – often monstrous tyrants, even crypto-terrorists – to win their trust and persuade them to make peace. The book is full of references to marathon late-night phone calls to warring parties, and the art, he says, is quite simple. You have to understand what drives them, appeal to their pride and vanity, and offer them a way out that saves face.

"Dignity is very important. It's about dignity. Without beating up on them, I say: 'If you're really a leader, and are interested in the interests of your people, you have to lead; you have to show courage. If you are really a strong man as you say you are, you have to look out for the weak in your society. Look at what you're doing to them.' They have to feel that you have their interests at heart. These are people with egos, and a sense of legacy, so you ask: 'What legacy are you going to leave?' Saddam probably saw himself as a modern-day Saladin – the glorious Arab warrior. He saw himself as someone who was building his nation. So you tell him: 'You want all this destroyed? How do you think you will be judged?'

"That matters, because these people often have a certain image of themselves, and they will go to an incredible extent to protect themselves. So I tell them what their actions mean, how they are seen by the outside world."

I suspect Annan must struggle to imagine how he will be judged himself, for his identity is so subsumed within that of the UN as to be almost indivisible. When I ask him to name the single greatest misconception about him, his first response is telling. "About me, or about the UN?" About him, I say. He falls into lengthy silent thought.

"That he's too soft," he offers eventually. "Because they feel he doesn't pound the table – not assertive enough. But it doesn't bother me, because sometimes you don't have to fight to get your way. You don't have to pick a fight to get them to change their mind, or get them to see things your way. You really don't."

Interventions by Kofi Annan is published by Allen Lane, rrp £25. To order a copy for £20 with free UK p&p, call 0330 333 6846 or visit guardian.co.uk/bookshop