During debates about who is the best boxer of all time, the former four-weight world champion Pernell “Sweet Pea” Whitaker is sometimes overlooked. He was not a fearsome puncher, nor was he a braggart who commanded headlines in the era before social media when newspaper reporters treasured outrageous quotes to sell a fight. But his peers always rated him among the sport’s elite.

Whitaker, who has died aged 55 after being hit by a car while crossing a road in his hometown of Virginia Beach, Virginia, was an Olympic gold medallist for the US at the Los Angeles games of 1984, after which he turned professional, first becoming a world champion as a lightweight in 1989. He eventually quit boxing in 2001, having fought in a total of 23 world title contests, of which he lost only three and drew one, in hugely controversial circumstances against the Mexican fighter Julio César Chávez.

One of his beaten opponents was the Scot Gary Jacobs, who challenged Whitaker for the WBC welterweight title in Atlantic City in August 1995. Jacobs had reigned as a British, Commonwealth and European champion but, like so many others, found Whitaker’s southpaw style and brilliant defensive skills almost impossible to fathom and he lost widely on points.

After learning of his old adversary’s death, Jacobs said: “If you want to learn boxing, watch his fights. His ringcraft was just incredible. He was so clever. He was almost impossible to hit cleanly. His skills were truly remarkable. People talk about great fighters, particularly at lightweight with men like Roberto Durán and Floyd Mayweather, but I think that he was the best of the lot.”

The former world heavyweight champion George Foreman said: “It was like watching a cat with boxing gloves. The best balance I ever saw in a boxer.”

Born in the tough Virginia coastal city of Norfolk, best known for its military bases, Pernell was one of seven children. His parents, Raymond and Novella, encouraged their son to pursue sports and he first went to a boxing gym aged eight, when it was quickly apparent that young Pernell had an aptitude for it. He was known to his family as Pete, after one of his uncles, and at an early boxing show, Whitaker’s mother had been calling out the name “Sweet Pete” only to see it misreported in a local newspaper as “Sweet Pea”. Ever thereafter, it seems, the name stuck.

Whitaker claimed to have had more than 500 contests as an amateur before his Olympic success. In 1984 he was captain of a US team that included Evander Holyfield, Mark Breland, Meldrick Taylor and Frank Tate, all medallists who would go on to become professional world champions.

Whitaker turned pro under the promotion and management of the Duva family, and was trained by the extrovert patriarch Lou. After a run of 15 straight wins, he lost on a dubious points decision in a world lightweight title challenge in France in 1988, but scored a convincing points victory over Greg Haugen to become the IBF lightweight champion the following year.

The win marked the beginning of Whitaker’s era of glory. By the time he fought Chávez in 1993, in front of a 60,000 crowd at the Alamodome in San Antonio, Texas, he had recorded 17 straight victories and had been a world champion at three weights – lightweight, super lightweight and welterweight. Defending the WBC welterweight title against a man who was undefeated in 87 professional contests, Whitaker was awarded a majority draw in an outrageous judging decision when virtually every ringside expert believed Whitaker had won clearly. Whitaker’s verdict was unequivocal: “I whipped his ass.”

In 1995 he took the WBA super welterweight title, but alcohol and drug problems blighted the later stages of his career. After losing his welterweight world title on points in another disputed decision to Oscar De La Hoya in 1997, he was banned from the sport after testing positive for cocaine, and lost again on points in 1999 to the Puerto Rican Félix Trinidad in his comeback for a final world title contest. In all, Whitaker won 40 pro fights, lost four and drew one with one no contest.

After retiring, he worked sporadically as a coach. He was sent to prison in 2003 for violating the terms of a probation order imposed for cocaine possession. As the fortune he earned in boxing was frittered away, he once more was in the headlines in 2014 when he won a court order to have his mother and two of his siblings evicted from a house he had bought 30 years earlier so that he could sell the property to cope with his mounting debts.

His marriage, to his childhood sweetheart, Rovanda Anthony, whom he wed in a boxing ring in Virginia Beach in 1985, ended in divorce. He is survived by his mother, and by three sons, Dominique, Dantavious and Devon, and a daughter, Tiara. Another son, Pernell Jr, predeceased him.

• Pernell Whitaker, boxer, born 2 January 1964; died 14 July 2019