(CNN) When the so-called "Fight of the Century" finally takes place in Las Vegas on Saturday, there will be great interest thousands of miles away in a rundown Soviet-style housing estate in eastern Europe.

If Floyd Mayweather can protect his unbeaten record against Manny Pacquiao, a Bulgarian man will experience a happiness he thought long-extinguished from his life.

Because, until recently, Serafim Todorov was boxing's forgotten man.

Mayweather vs. Pacquiao is expected to be the richest boxing match in history -- some tickets are on sale for nearly $130,000 -- while Todorov, known simply as "the man who beat Mayweather," hopes to set up a small gym by the Black Sea.

But Todorov lost the final, claiming a silver medal while Mayweather signed up with the U.S. promoters, catapulting his career forward. He is now undefeated in 47 professional fights ahead of his long-awaited clash with Manny Pacquiao on May 2.

But Todorov lost the final, claiming a silver medal while Mayweather signed up with the U.S. promoters, catapulting his career forward. He is now undefeated in 47 professional fights ahead of his long-awaited clash with Manny Pacquiao on May 2.

Todorov (right) turned them down -- a decision he regrets to this day -- being confident he would win gold and have a successful amateur career. He beat the 19-year-old Mayweather by one point in a controversial featherweight semifinal, leaving just Thailand's Somluck Kamsing (left) as an obstacle to Olympic glory.

Todorov (right) turned them down -- a decision he regrets to this day -- being confident he would win gold and have a successful amateur career. He beat the 19-year-old Mayweather by one point in a controversial featherweight semifinal, leaving just Thailand's Somluck Kamsing (left) as an obstacle to Olympic glory.

The 45-year-old believes things could have turned out differently. There was a life-changing moment at Atlanta's 1996 Olympics when American boxing promoters approached him with a lucrative professional contract.

The 45-year-old believes things could have turned out differently. There was a life-changing moment at Atlanta's 1996 Olympics when American boxing promoters approached him with a lucrative professional contract.

Pictured here showing CNN's Alex Thomas his old photo collection, the former boxer won three world amateur championships -- in 1991 at bantamweight, and in 1993 and 1995 at featherweight.

Pictured here showing CNN's Alex Thomas his old photo collection, the former boxer won three world amateur championships -- in 1991 at bantamweight, and in 1993 and 1995 at featherweight.

Former Olympic silver medalist Serafim Todorov lives off state handouts of under $500 a month -- a far cry from the lavish lifestyle of "Money" Mayweather, who has splashed out on $25,000 mouthguards ahead of his blockbuster bout with Manny Pacquiao.

The last man to defeat Floyd Mayweather in the boxing ring can be found in this gym in the small Bulgarian town of Pazardzhik.

The last man to defeat Floyd Mayweather in the boxing ring can be found in this gym in the small Bulgarian town of Pazardzhik.

Now, as global attention focuses on the richest fight in the history of the sport, the sad story of the last boxer to defeat Mayweather -- almost two decades ago -- has come to light.

The American will pocket a substantial part of the reported $300 million purse regardless of the outcome against Pacquiao. Todorov, meanwhile, would have to wait more than 35,000 years on his current monthly pension of under $500 to earn such a fee.

But although he harbors great regrets and bitterness over the way his life has unfolded, none of it is aimed towards Mayweather.

"I congratulate Floyd because everything he achieved in his life, he achieved with hard work," Todorov, who ended a teenage Mayweather's hopes of winning gold at the 1996 Olympic Games in Atlanta, tells CNN.

"I don't envy him because he did the right thing, he worked hard for everything he achieved in his career. He deserves everything he has."

Turning point

That loss on August 2, 1996 has gone down in the history books as a turning point in Mayweather's career.

A controversial decision that evening gave Todorov a 10-9 points victory, putting the Bulgarian into the featherweight division final.

Since then, there has been no looking back for Mayweather, who is undefeated over 47 professional bouts and is regularly listed among the world's wealthiest athletes. He has won titles in five weight divisions and taken home at least $25 million for each of his past 10 fights, according to Forbes.

JUST WATCHED Pacquiao talks to CNN about Mayweather showdown Replay More Videos ... MUST WATCH Pacquiao talks to CNN about Mayweather showdown 03:56

"I'm happy with how the result went. I'm really happy because that made me strive and work hard to get to where I'm at today," Mayweather said of his Atlanta loss on Showtime's "Inside Mayweather v Pacquiao."

Yet while Mayweather's career rapidly ascended on an upward trajectory, Todorov's descended almost as dramatically in the opposite direction -- starting with his next fight.

"At the gold medal match I felt like the judges had been corrupted and were involved in some sort of scheme," Todorov says of his defeat to Thailand's Somluck Kamsing in the Olympic final.

"I had beaten my opponent countless times before with only five days of training, and after that match everything went downhill in my life."

As Mayweather credits events of the 1996 Olympics in getting him to where he is today, so too does Todorov look at his time in Atlanta as being a turning point in his own life.

"At that point all I ever wanted was to have a gold medal from the Olympic Games, which would have given me the chance to consider myself as the perfect boxer," he says.

"After I lost we had to stay in Atlanta for two more days to wait for our plane. I drowned my sorrows in alcohol and when I came back to Bulgaria, I wanted to quit boxing."

Regrets

Todorov's journey to the present day has not been an easy one -- a life plagued by depression brought on by a number of broken promises and wrong decisions.

The choice he made in the immediate aftermath of his victory over Mayweather still haunts him today.

Then 27, he was taken to doping control along with his opponent when he was approached by a group of American boxing promoters.

On the verge of winning Olympic gold, and with three world amateur titles already to his name, he was offered a lucrative deal to turn professional -- but turned it down with little thought.

JUST WATCHED Evander Holyfield 'excited' about Mayweather-Pacquiao Replay More Videos ... MUST WATCH Evander Holyfield 'excited' about Mayweather-Pacquiao 03:25

Todorov believed biding his time was the more sensible option, preferring to continue with his successful amateur career and hope he would soon emerge as the new Olympic champion.

Mayweather, meanwhile, signed up with the same promoters and the rest is history.

"I was 100% sure I would win the gold medal and return in Bulgaria as a champion," Todorov says.

"If I were there right now, I would tell myself to sign the contract and stay in America. If I knew that my life would go downhill from there, I would have signed the contract right away.

"I felt like I was out of luck because I turned down a great opportunity and after that I had no choice but to return to Bulgaria where my depression started," he adds. "In my opinion the best place to be a professional boxer is America, and I lost my chance."

The end



Upon his return home, promises of extra financial support from the Bulgarian Boxing Federation following his Olympic exploits failed to materialize.

Such treatment led Todorov, struggling to support his family, to try to switch nationalities and move to Turkey, whose boxing federation prepared to offer him a larger salary to represent the nation at the 1997 world championships.

A disagreement between the two federations, however, soon brought him crashing back to square one -- the politics of boxing had stifled him once more, this time proving to be one step too far.

JUST WATCHED The "Billionaire Boys" Fight Club Replay More Videos ... MUST WATCH The "Billionaire Boys" Fight Club 02:29

Todorov quit amateur boxing after refusing to represent Bulgaria and, bar a handful of professional fights arranged for some much-needed cash, his career was over by 2003.

"My life now is my family, I don't really talk to people anymore," says Todorov, whose Olympic silver medal was lost after being donated to a museum.

"I've suffered from depression countless times in my life and even now I'm depressed. I just live a mundane life with my family."

All is not lost

Beating Mayweather may not have brought with it all the fruits that Todorov might have imagined, but the 45-year-old, now living in the small Bulgarian city of Pazardzhik, is at least starting to reap some of the rewards.

JUST WATCHED Pacquiao: Faith is important, not earthly things Replay More Videos ... MUST WATCH Pacquiao: Faith is important, not earthly things 03:50

"The whole world knows about me now. I feared they might have forgotten me. Before when I was still boxing people talked about me because of my titles, and now I'm famous because of Floyd," says Todorov, who has been offered the chance to set up a boxing gym bearing his name by the Black Sea.

"Floyd, I congratulate you for being undefeated and I hope you stay undefeated. I would feel really proud because I will be the only man that has ever defeated you."