On a cold and grey afternoon in Washington DC the jubilation that swept through the streets a week ago has all but gone. The screaming crowds and honking cars that engulfed the city last Tuesday night, confirming Barack Obama's electoral victory, have been replaced by a drone of traffic and the throaty hum of Joe Frazier singing to himself in an anonymous hotel room.

"I ever told you I was a great singer?" the old fighter eventually asks. Three giant rings glitter on his gnarled fingers as Smokin' Joe, a heavyweight crooner with the blues in his bones, looks up and whoops: "I'm still smokin', man!"

It might also be said on bleaker days, when Frazier is alone at home in Philadelphia with only his haunted memories for company, that a mere wisp of smoke still rises from the ashes of his legacy. Frazier's immense achievement as world heavyweight champion, at a time when boxing carried such sporting and political resonance, was torched by Muhammad Ali's jibes and taunts.

As easy as it is to love Ali, and especially his enduring image as the bravest and funniest and most significant sportsman in living memory, he was unspeakably cruel to his bitterest rival. Demeaning Frazier as "flat-nosed" and "backward", as a gorilla and an Uncle Tom, Ali's banter was soured with malevolence. Having been so scorched and trashed in the past it is little wonder that Frazier's name did not appear among the black pioneers exalted last week alongside Obama.

Ali himself, meanwhile, was celebrated again as the black American who, after Martin Luther King, did most to confront racial prejudice in a once seething country. "I lived with it for years," Frazier shrugs at his neglect. "But I like Obama. I think they picked a fine guy in him. Listening to him speak, it sounds like he's going to be fair and clearcut."

The bad blood between Ali and Frazier is a darker and more tangled business. But here in Washington, at the start of a new era for America, Frazier offers up a reminder of how he and Ali were once friends. "I helped him out," Frazier says. "I felt sorry for him in a way."

In 1967, having embraced Islam and changed his name from Cassius Clay, Muhammad Ali refused to fight in the Vietnam war. "I will face machine-gun fire before denouncing Elijah Muhammad and the religion of Islam," he insisted. "I'm ready to die." As the reigning heavyweight champion of the world, and at the height of his dazzling powers, Ali was stripped of both his title and his licence to box.

Broke and vilified, Ali started calling Frazier, his eventual replacement as world champion. "He'd be phoning every other day to say, 'You got my title, man! You got to let me fight you!'" As he repeats that plea Frazier slips into an impersonation which sounds less like Ali in his fast-talking pomp than his old foe after Parkinson's disease had made his speech slurred and halting. "I said, 'OK, I'll see what I can do.'

"I went to see President Nixon at the White House. It wasn't difficult to get a meeting because I was heavyweight champion of the world. So I came to Washington and walked around the garden with Nixon, his wife and daughter. I said: 'I want you to give Ali his licence back. I want to beat him up for you.' Nixon said, 'Sure, I'd like that.' He knew what he was doing and so Ali got his licence back."

Ali had been in exile from the ring for three years before Frazier's intervention in 1970. Did Ali thank him? "I don't remember. Maybe he did - but I doubt it. I was just happy he got his licence back so I could clean him out."

Frazier had also given Ali money, but that did not stop the sudden animosity which welled up in the returning hero. Ali was initially amusing. "Joe Frazier is too ugly to be champ. He can't talk. He can't box. He can't dance. He can't do no shuffle and he writes no poems."

But the joking soon stopped. "Joe Frazier is an Uncle Tom," Ali ranted. "He works for the enemy."

Joe's son, Marvis, winces on the sofa opposite his father and me. "I used to get beat up every day at school by guys who would say, 'Your dad's a Tom'. It was terrible."

A compelling new documentary, The Thrilla in Manila, is unflinching in the way it documents the systematic racial abuse Ali directed against Frazier for the next five years - culminating in the final fight of their epic trilogy in 1975. They had each won one bout and Ali demeaned Frazier as a big black gorilla who communicated in grunts rather than words.

Frazier looks at me wearily when I ask him about such ridicule. "I know my destiny. I was born into animosity, bigotry and hatred. We had water for white folks, and water for coloured folks. White lines, black lines. I came from Beaufort in South Carolina and it was tougher than Georgia, Alabama and Mississippi."

Where Ali lived in Louisville, Kentucky, and had a comparatively sheltered life, Frazier's parents were impoverished sharecroppers. "I was one of 14 [children] and I had to work on the land from when I was seven. My parents really suffered back in them days."

Frazier eventually took the Greyhound bus, "the dog", from the South to Harlem. He ended up working in a Philadelphia slaughterhouse. "I was the drain man. My job was to make sure the blood went down the drain. But sometimes, early in the morning, I'd go down that long rail of meat and work on my punching. That's how [Sylvester] Stallone got the same idea for Rocky - just like he used the story about me training by running up the steps of the museum in Philly. But he never paid me for none of my past. I only got paid for a walk-on part. Rocky is a sad story for me."

Ali, however, is a more lasting adversary. In later years Frazier's resentment was such that, when asked for his opinion about Ali being chosen to light the Olympic torch, Smokin' Joe said: "I think he should be pushed into the flames." He also wrote in his autobiography that, "I'd like to rumble with that sucker again, beat him up piece by piece and mail him back to Jesus".

Considering the brutality of their last fight in Manila, and suggesting that it may have contributed to the onset of Ali's Parkinson's, Frazier says now: "I clearly won. The proof is in the pudding. I'm here talking, walking. I'm 64 and, yeah, I'm still having fun. I hope the Lord will forgive him. I don't care who you are - whatever you done as a young man it comes to bite you in the butt. Trust me. Sometimes God comes down and puts his hand on you if you're too big in your thoughts."

It is impossible to know how much damage Manila might have done. But Ali said it was "the closest thing to dying" - while Frazier, who had beaten up his enemy remorselessly, was plunged into near darkness when his only good eye was sealed shut in the last few rounds.

"My left eye went when I was young. I was working the speed bag and some steel went in the eye and scratched it to pieces. I was kinda blind in that eye."

Frazier laughs when asked how he could have fought anyone, let alone beaten Ali, with only one eye. "Some doctors were my best friends. We sheltered the story and kept it a secret. And I learnt the eye-chart by heart."

Yet in Manila, towards the end, Frazier could not see much out of either eye. Slumped on his stool before the 15th and deciding round, he protested angrily when his trainer, Eddie Futch, would not allow him to answer the bell. "I was begging: 'Eddie, please, let me continue. Don't you stop this fight'."

In the opposite corner Ali wanted to quit. "Cut 'em off," he said mournfully, looking down at his gloves, unable to bear another round of torment.

But Futch had already stopped the greatest fight in heavyweight history. Ali collapsed in exhaustion while Frazier raged blindly. Even now Frazier does not hesitate when asked whether he would have been willing to risk his life in that last round. "Yeah."

After they had been taken back to their dressing rooms Ali called for Joe's son. Marvis, who was only 15 then, smiles sadly at the memory. "[Ali] said, 'I just want to apologise to you, your family and your father for all I said. You tell your father'. I did."

Joe looks up quietly. "I said, 'Hey, son, why's he saying that to you? Why's he not apologising to my face?'"

Darkness has descended across Washington; but Frazier's hotel room is ablaze with light. As the photographer drapes a white sheet over the frame of the four-poster bed, Smokin' Joe whoops again. "You got some ladies behind there for me?"

"Yeah, hon," she laughs, "you come over here." Joe squeezes my hand hard and winks. "Now we talkin'."

In front of the camera Frazier goes into a low crouch, bobbing and weaving, and throwing punches at thin air. "Bang, bang, boom!" he grunts - reliving the moment when, in his first fight with Ali at Madison Square Garden in 1971, he knocked down his then unbeaten opponent to clinch a momentous victory.

"That was the best fight between us. And it was harder than Manila because he was a much better fighter then. But I dusted him off."

Frazier keeps throwing punches for the next five minutes and, as his breath becomes ragged, we suggest a break. But Smokin' Joe can't stay still for long. He drags out an ancient cassette recorder and plays himself singing a version of My Way - with revised lyrics, full of pugilistic references, especially written for him by Paul Anka.

Smokin' Joe, clearly, is having fun - but what about Ali, lost in his silent cocoon? "He hollered a lot but, hey dog, it's more than 30 years now. I don't hate anyone. Let him try and put his life back together. We done with fighting. What do we need to be angry about? We been through enough wars."

Frazier reaches for a black hat, plonks it on his head, and looks up at the photographer. "OK, lady," he chortles, "how do you want me?"

"I want you just as you are, Joe," she says, "still smokin'!"

"You got it," the great old fighter says as he starts shadow-boxing again, as if shedding the last ghosts of his past.

"Here we go ... bang, bang, boom! Bang, bang, boom!"

· True Stories: Thrilla in Manila will be screened on More 4 at 10pm tonight

When they were kings

Mar 1971 Madison Square Garden

Frazier wins by unanimous decision

This was Ali's first title shot since regaining his licence and billed simply as 'The Fight'. Ali starts well but is soon slowed down by Frazier's left hooks, one of which knocks him down in the 15th. The challenger gets up but cannot force his way back in the contest

Jan 1974 Madison Square Garden

Ali wins by unanimous decision

A non-title fight that Ali dominates. He knocks Frazier down in the second round only for the referee, Tony Perez, to step in and incorrectly rule the round was over. But Ali keeps up the pressure to secure victory

Oct 1975 Manila

Frazier retires before the 15th round

'The closest thing to dying that I know of,' says Ali, who dominates for three rounds but wants to quit by the 10th. But the champion continues and by the 14th, it is Frazier who is the beaten man. His coach, Eddie Futch, refuses to send him out for the 15th