Jimmy Carter has snubbed the lecture circuit cash cow to devote his post-presidential life to brokering peace in the Middle East and fighting disease. So why is it that so many Americans can't stand him? Jonathan Freedland asks him

Funny business, being a former president of the United States. People still use your old title - you are always Mr President - and there's the thrill of belonging to the world's most exclusive club. Current total membership: three, though a fourth member, George W Bush, will be inducted on January 20 next year.

The man who has been in the club the longest - since his ejection from the White House in 1980 - has also rewritten its rules. True, there was not much in the way of precedent to guide Jimmy Carter. Franklyn Roosevelt and John F Kennedy died in office, Lyndon Johnson very soon after leaving it; Richard Nixon retreated into a kind of private penance. But if there was a pattern for the post-presidential life, it surely involved the lecture circuit, the corporate boardroom, a dose of charity work and, most visibly, the golf course. The template was set by Carter's immediate predecessor, Gerald Ford, and has been honoured since then by George HW Bush. If the current president departs radically from that pattern, it will be a surprise.

Carter is different. In what he concedes was "a somewhat weak moment", he vowed to reporters soon after his defeat by Ronald Reagan - a defeat that, despite the polls foretelling it, came as a terrible shock to Carter - that he would not be treading down the familiar, profit-making route. "Others have wanted to make a lot of money," he says of his fellow ex-presidents. "Which I don't criticise. They make enormous sums of money, as has been recently revealed about the former president." (He is referring to Bill Clinton, whose joint tax returns with Hillary showed an income of $109m since leaving the White House.) "That doesn't appeal to me," he says, adding that he gets "a good income" from his book sales. He's just completed a 25th, a portrait of his mother Lillian, and announces that "they've all done quite well, sold hundreds of thousands of copies".

Carter's chief output, though, has been in the realm of good works. The result is the Carter Center, originally a one-man mission, now the employer of 150 people, with a restless, global brief that ranges from ridding the world of "neglected diseases" to monitoring disputed elections - with a little light Middle East peacemaking thrown in. This is no faceless NGO. The heart of it is still the personal diplomacy - or ceaseless meddling, according to his enemies - of Carter himself.

This post-presidential work bagged Carter the Nobel Peace prize in 2002 and has led even his critics to call him the most successful ex-incumbent of the White House in American history. He has devised a new model, the activist ex-presidency, which appears to have inspired, at least partially, Clinton in his global foundation and high-profile effort to combat Aids.

But it also means Carter has deprived himself of the most obvious perk of former office: the cosy, consensual place in public affections reserved for an above-the-fray former statesman. Eighty-three years old, he should by now be entitled to national treasure status. Instead, Jimmy Carter remains as controversial a figure today as he was when he sat in the Oval Office. He is denounced on op-ed pages and in the blogosphere, even as armies of activists regard him as perhaps their most prestigious advocate. On at least one of the major international questions of our time, he remains a player - and no one ever said that about Jerry Ford.

These past three decades of work - too long to be the twilight of Jimmy Carter - also raise a fascinating question about politics itself: why is it that those slammed, whether for weakness or shabby compromises in office, so often flourish once outside it? Why is it that Al Gore was able to become the world's leading advocate of action on climate change only after he turned his back on electoral politics? Why is it that Carter looms larger the further behind him the White House recedes?

Jimmy Carter cuts a modest figure in person. He walks with a slight stoop and projects little of the backlit radiance that never seems to leave the most charismatic politicians. Together with his wife Rosalynn, who is rarely far from his side - he in tweed jacket, she nursing a nice cup of tea and a slice of cake - they could be a couple of vacationing pensioners from rural Georgia, except, of course, for the extensive Secret Service detail hovering close by. They are on the Welsh borders for the Hay book festival, but they will also make time to stay with friends and do some sightseeing related to Dylan Thomas, whom Carter rates as the greatest poet of the last century.

Later that night, he will prove a smash at Hay, delighting a packed marquee with a fluency, cogency and mastery of detail that is not only mightily impressive in a man of his age, but that the audience cannot help but tacitly contrast with the current resident of 1600 Pennsylvania Avenue. It helps that Carter's message is so in tune with liberal, European sensibilities. In one riff, he offers that the next president would not need 100 days to change America's image in the world, just "10 minutes" - the time it would take to deliver an inaugural address that would promise that America "will never again torture a prisoner... never again attack another country unless our security is directly threatened", will honour its international agreements and do the right thing on climate change.

Carter's assessment of his own record in office fits that message. He took some "revolutionary" steps, he says, and he is proud of them. "We established human rights as the basis of our foreign policy, whereas in the past our government had been in bed with every dictator on earth if they supported our economic framework. We normalised diplomatic relations with China. We brought peace to the Middle East, between Israel and Egypt. We kept the peace with the Soviet Union. We told the truth. We kept our country at peace, we never dropped a bomb, we never launched a missile."

All of which builds the only leverage Carter says he has: "I have moral authority - as long as I don't destroy it." It also means he carries himself more as a spiritual leader than a political one; more Desmond Tutu than George Bush Snr. It helps that Carter is a man of deep faith - he is often described as the first born-again Christian to serve as president, and still teaches a Bible class every Sunday morning in his hometown of Plains, Georgia. The fact that the Carter Center can legitimately claim to have mediated successfully in a clutch of armed conflicts and to have all but rid the world of the hideous Guinea worm disease - there were 3.6m cases of it when the Center started work and just 5,000 now - only reinforces the image of a tireless agent for good, even, in one journalist's description, a "living saint".

So why is it that so many Americans can't stand him? The answer lies partly in the very things that non-American audiences admire. Where Europeans see a dove, Carter's detractors see a weak president who delighted a hostile world only too happy for America to walk small. They associate Carter with America's humiliation in Iran - thanks to both the 444 days in which 52 US diplomats were held hostage in Ayatollah Khomeini's Tehran and the abortive attempt to rescue them - as well as a stalled economy. For them, the Carter era represented a crisis of American self-confidence, which took eight years of Reagan's sunny optimism to restore.

Europeans may warm to Carter's clear pride in a clean sheet on the use of force - he branded the Iran hostage rescue effort a humanitarian, rather than a military, mission so as to maintain the purity of his record - but too many Americans saw that as wimpishness. (The contrast with George Bush, revelling in his status as a "war president", could not be sharper.)

As to Carter's later work, its very goodness seems to get under many Americans' skin. They found him preachy as president, and still do. The former editor of the New York Times, Joseph Lelyveld, put it succinctly when he wrote last year: "The former president's peculiar combination of rectitude and starchy pride can be a little irritating, as it was three decades ago when he lectured us on energy independence and then blamed our 'malaise' on our failure to heed him."

Even Carter's prodigious knowledge, his phenomenal recall of data, can count against him. When Saturday Night Live, America's weekly satirical TV show, sent up Carter back in the 70s, they would depict him answering questions on a radio phone-in show. Callers would ask the president about, say, their dodgy boiler: "Now turn the red valve clockwise..." the president would advise, a micro-managing know-all.

I wonder if all these years of castigation and derision - "the Carter administration" is used as a synonym for failure in the way "the Major government" is deployed here - are partly what drives the former president to keep pushing himself so hard when most men in his position would be heading for the putting green. Is his work fuelled by the urge to repair his reputation; is it a quest for redemption?

"I honestly don't have that feeling at all," he says. For one thing, he rejects the premise of the question. Though he "can't deny" that his ex-presidency has been more of a success than his presidency, he still vigorously defends his term of office. The motivation for the Carter Center was simple, he says: "We had an unprecedented idea." (Like most American politicians, he deploys a royal "we", probably to avoid the more obvious narcissism of "I", but still a formulation that suggests an ego in good health). There were desperate problems facing the world and where no one else was acting, where there was a "vacuum", the Carter Center would step in. A politician who had struggled to be popular would make a virtue of championing unpopular causes, especially those thoroughly ignored by everyone else. So Carter works on river blindness or increasing the yield of food grains in Africa - issues which, to a serving politician, would spell "no votes" in capital letters.

Carter rejects the redemption notion for another reason, too. "I don't consider it to be a sacrifice to go to Nepal or to talk to Hamas. It's exciting, challenging, unpredictable, gratifying, adventurous. It's a nice thing to do."

The only hint the former president gives that there is a connection between his current labours and the fate of his presidency comes when I ask if he'd be doing all this if he had somehow defied the odds and beaten Reagan in 1980. "I've thought about that a good bit, and I don't think I would have," he says. Is that because he would have regarded his work as complete after two full terms, that he would have felt fulfilled? No. It's because "if I'd had another term, peace in the Middle East would have become permanent. That's my honest opinion. It would have been done". And therefore that part of the Carter post-presidential workload - acting as unofficial would-be peacemaker between Israel and the Arab world - would not have been necessary.

We have come to the most vexed aspect of Carter's current role in the world, the activism that ensures he remains a focus of anger and political contention rather than warm-bath, humanitarian consensus. In 2007, Carter authored a polemic titled, without punctuation, Palestine Peace Not Apartheid, thereby becoming the weightiest world figure ever to have made the comparison between Israel and apartheid South Africa.

For many pro-Palestinian campaigners, it was a relief to hear, at last, a mainstream American figure speak so robustly about the plight of those living under occupation, recently denouncing, for example, the "imprisonment" of the people of Gaza was "one of the greatest human rights crimes on earth". But, once again, the reaction to Carter in the US is rather different. The apartheid comparison was seen by many as a step too far, conferring on Israel an illegitimacy that is somehow final, as if it could be remedied only by Israel's dissolution, much like the South African regime's in 1994.

In our interview, Carter was at pains to clarify that the book title refers not to Israel itself - in which Palestinians are citizens, have the vote and so on - but solely to the Palestine of the occupied territories. Curiously, the book does not seek to make a full case even for this limited use of the apartheid analogy, discussing it in only four paragraphs across the whole text. Some critics reckon Carter deployed the "a" word solely to ...#8594; ensure attention for his book. If so, it certainly worked, offering yet another proof that Carter, for all the remarkable good works, remains a politician, one who, in his day, knew how to throw a punch.

But the book confirmed the suspicions that many Israelis and Jewish Americans had long harboured against Carter: that though he might say all the right things - condemning suicide bombings against Israeli civilians, insisting that his primary aim in 32 years of Middle Eastern engagement has been "to bring security and peace to Israel" - he feels genuine compassion for only one of the two sides. Other, more neutral observers say he remains too credulous in all spheres, and especially when it comes to those who mean Israel great harm.

All this came to a head in April, when Carter embarked on a round of high-profile shuttle diplomacy, meeting Syria's president and, more controversially, the leadership of Hamas, an organisation proscribed as terrorist not only by Israel but by the US and EU, too. He was roundly condemned at home, even by leading Democrats, and when he got to Israel, bearing messages from those he had met, no frontline politician would so much as meet him. The liberal Haaretz newspaper, which argued that Carter should be treated like "royalty" simply for having negotiated the 1979 peace treaty with Egypt that did more than any other event in the country's history to secure Israel's place in the region, was a lone voice.

Had the former president seen, either in Syria's Bashar al-Assad or the Hamas leaders, what he had seen 30 years ago in Egypt's Anwar Sadat - a man ready to make peace with Israel? His answer goes some way to explaining why Carter still infuriates his detractors. "I didn't see it, but I saw the potential of it," he says, describing hours of talks that went late into the night. "The head of Hamas, Khaled Mashal, is a physicist, highly educated. Obviously shrewd politically, since he orchestrated a very successful campaign with a glorious victory in 2006. The deputy head of the politburo is a cardiologist, distinguished in his field... There are physicists, medical doctors and others who have also shown political acumen, competence and boldness and have within them the capability... to make an option of peace."

Note that Carter does not simply say that Hamas have an electoral mandate, that they represent part of the Palestinian people and therefore have to be engaged with. He goes further, praising them, speaking of their "glorious" election victory or suggesting their backgrounds in medicine makes them men of peace (thereby giving his enemies an easy shot, as they point out that, for example, Ayman al-Zawahiri, deputy to Osama bin Laden, is himself an eye surgeon). When the former president praises Assad's "intelligence, competence and strength" or speaks of Hamas's "superb" record in municipal government, he walks into the same trap: he is no longer making the case for dialogue with Assad and Hamas. He is making the case for Assad and Hamas.

Carter is aware of this danger. I ask whether he detects an in-the-gut conversion to the path of peace in the Hamas leaders he met: "I do, but I may be naive and I may be mistaken."

What about Hamas's founding charter, replete with antisemitic imagery, some of it drawn from the notorious Tsarist forgery, the Protocols of the Elders of Zion? "It's terrible," he says. "I despise antisemitism in anyone. I think anti-Jew is a violation of the basic principles on which my life is built. I grew up with it in the south with anti-black, so I saw the ravages of racial discrimination." But when the charter was raised in those talks with Hamas's top echelon: "They ridiculed it as being ancient, passé, an inconsequential document. But I don't speak for anyone else."

Carter's mission to Hamas also badly discomforted Barack Obama. When we met, the primary process was still unfolding and the former president had made no formal endorsement of a Democratic candidate, but he had left pretty heavy hints that he favoured Obama. So when Carter went to Damascus, Obama was forced to distance himself. "It gave [John] McCain a stick to beat Obama," one senior Democrat told me. "Some help."

That same Democrat sees it as part of an ingrained Carter habit. "He is indifferent to political timing, part of his self-image as holy man, but often undermining the causes he pursues." That, and the lingering association with defeat, mean Carter does not quite enjoy the party elder status among Democrats that you might expect of a former president. At Bill Clinton's second nominating convention in 1996, for example, Carter was not even present, still less an honoured guest.

But his words still make news. He tells me that he believes that "inherently, John McCain is a more warlike leader than George W Bush". McCain would be better than the current president on issues of torture and climate change, but, says Carter, "He has even said in an offhand but I think sincere way that... American troops might be [in Iraq] 100 years from now. I don't think Bush has gone that far. And I don't think Bush has taken any step - in going into Iraq or staying there or increasing the number of our troops - where McCain didn't say he should have done more."

Carter gives a backhanded compliment to the Clintons when he says Obama has been "greatly strengthened... as a campaigner because he's had to face Bill and Hillary Clinton." He's been "battle-hardened". What, then, of an Obama-Clinton dream ticket? "It would be the worst mistake that could be made. That would just accumulate the negative aspects of both candidates." He cites the polls showing 50% of US voters with a negative view of Clinton. "If you take that 50% who just don't want to vote for Clinton and add it to whatever element there might be who don't think Obama is white enough or old enough or experienced enough or because he's got a middle name that sounds Arab, you could have the worst of both worlds."

Has he been disappointed by the hardball campaigning of his fellow ex-Prez, Bill Clinton? "I think it has hurt him," he says, volunteering as evidence Clinton's comparison of Obama's victory in the South Carolina primary to that of Jesse Jackson in 1988, which many took to be an attempt to cast Obama narrowly as the black candidate. "The next day Bill Clinton called me and told me that was not his intention and I believed him. I don't think he deliberately meant to make a racist remark." But having equated Obama to Jackson, says Carter, it was obvious that Clinton's words would be interpreted that way. Has it caused long-term damage to Clinton's reputation, even in the eyes of history? "It may have."

Obama supporters will cheer that; Clintonites will see it as typical Carter sanctimony. Division and argument still follow Carter, even into his ninth decade. Will he be given a role befitting his status at the Democratic convention in Denver in August? The answer to that will tell us much about the current standing of this former US president - and quite a lot about the person who would succeed him.