Want to listen to this article out loud? Hear it on Slate Voice.

The writer Stanley Crouch has a great line about Muhammad Ali. When Ali came to fame in the 1960s, he was a grizzly bear, fierce and uncontrollable, Crouch told Ishmael Reed in an interview. In the second act of his career, Ali was a circus bear, still dangerous but also entertaining. In his last act, when he no longer seemed threatening, Ali was a teddy bear, soft and comforting and silent.

It’s Ali the grizzly bear who is frequently invoked as the country debates whether athletes like Colin Kaepernick should protest police violence on a national stage. But in hailing Ali for carving the path for those like Kaepernick, we often forget the high price the boxer paid for his political views. We forget that Ali had to be declawed to be beloved.

When Cassius Clay joined the Nation of Islam and dropped what he considered his slave name to become Muhammad Ali, he was attacked by politicians and the media. “I pity Clay and abhor what he represents,” wrote Jimmy Cannon. Even Martin Luther King Jr. said Ali “should spend more time proving his boxing skill and do less talking.”

When he refused to fight in the U.S. military in Vietnam, he was convicted of draft evasion and sentenced to five years in prison. He remained free while appealing the conviction, but boxing officials stripped him of the heavyweight championship and his license to box. He was called a coward, a traitor, a dupe, and an uppity N-word.

During all this, Ali stood almost completely alone.

In 1967, a group of America’s top black athletes including Jim Brown and Bill Russell met with Ali in Cleveland. Legend has it they gathered to show support for Ali. But I found evidence to the contrary in researching my Ali biography. Those other black athletes were really gathered in an attempt to convince Ali to make a deal with the government and get back to boxing, having been offered a financial incentive by one of Ali’s white boxing promoters to do so. Only when Ali refused, saying he’d die before compromising, did the athletes issue a statement of support for their peer.

Ali lost three-and-a-half prime years and millions of dollars. When the Supreme Court overturned his conviction, he said he wouldn’t sue for damages. He’d done what he thought was right, Ali said, and the government lawyers did what they thought was right. Everyone was entitled to his own view.

Ali began to win public favor when he accepted his conviction and moved on without griping. He gained more support when he returned to the ring, clearly diminished as a fighter, and lost in a brilliant and brutal bout with Joe Frazier in 1971.

He fought Frazier a second time and won. He regained his championship. He stopped talking quite so much about race and politics. And then he began to get old. His voice slowed. He gained weight. He lost more fights. He retired. He got sick. He lit the Olympic torch in Atlanta in 1996, his hands shaking from Parkinson’s, letting the world see him in all his imperfection, which took another kind of courage.

It’s troubling, though, that Ali became more lovable as he became more harmless. It’s backward. When we reflect on Ali’s life, we should remember him most for the occasions in which he most infuriated us. That’s when he was bravest, and that’s when he showed us what kind of country we could be if we were willing to fight.

Athletes who have spoken out for change, like Kaepernick and LeBron James, should be admired, not mocked, for their courage. The people seeking to shut them up are the same sort of people who refused to recognize Muhammad Ali’s right to cut his symbolic ties to slavery by changing his name.

Some say Kaepernick would be playing in the NFL today if he were a better quarterback, but history and human nature suggests that’s not true. Muhammad Ali was the greatest boxer in the world, and he was still pilloried by the public, stripped of his title, and exiled. The racist attacks on these men come from a source that doesn’t care how talented or successful a black athlete is.

If we expect sports heroes to be bold and brave on the field but passive and compliant off it, that’s our problem, not theirs.