Bill O’Reilly’s fall from grace at Fox News reached its end with his ouster on Wednesday, but the trouble started more than a decade ago. The first public sexual-harassment suit against the now ex-host came in 2004 when former employee Andrea Mackris claimed he’d subjected her to lewd phone calls and threatened retaliation if she reported him. Over the years, O’Reilly and Fox News spent at least $13 million settling claims that he’d mistreated female employees. Meanwhile, he sat behind his desk and broadcast his views to millions, calling his female co-host “eye candy” and a woman who was raped and killed “moronic” for wearing a miniskirt.

This 13-year history makes the particular moment of his firing seem rather arbitrary. The pressure began to rise last summer when Andrea Tantaros sued Fox News for sex discrimination, accusing both O’Reilly and then–recently deposed chairman Roger Ailes of unwanted sexual overtures that she repeatedly reported to workplace officials. In January, the New York Times uncovered a settlement Fox paid to former host Juliet Huddy to keep quiet about O’Reilly’s alleged harassment. But it wasn’t until the Times published details of a few more women’s allegations and settlements in early April that the turning tide of public opinion, and a campaign to make advertisers pull support, forced Fox News to drop the guy for good.

It’s easy to pick out similar turning points for O’Reilly’s peers, the other powerful men who’ve recently been undone by mounting allegations of sexual assault and harassment. Ailes’ moment came last summer after a lawsuit filed by former host Gretchen Carlson re-upped old claims against him and brought previously unknown victims of Ailes’ alleged mistreatment out of the shadows. Bill Cosby owed his final journey into exile to comedian Hannibal Buress, who reminded a Philadelphia audience in October 2014 that more than a dozen women had accused the famous moralizer of sexual assault, spurring dozens more women to come forward.

For Donald Trump, too, there was a clincher: the Access Hollywood tape with his famous “grab ’em by the pussy” line leaked a month before the election. Though a few women had made allegations against Trump before, many more went public after the tape’s release with stories of Trump doing exactly what he’d said he’d done. For a moment, as even party loyalists recoiled from this man caught on tape bragging about sexual assault, it seemed Trump might have come to the end of the road. But the road kept going. A few weeks later, America elected him president.

All these men had long histories of allegedly abusing women. None were deposed from their seats of prominence when those histories first came to light; they were dropped when external forces, including the indignation of the public, made their misdeeds too costly to ignore. Is there some tipping point, some number of victims or degree of severity before which abominable behavior can be excused—and past which no sexual predator is able to keep his job? If so, where does that threshold lie, and what does it take to turn public anger into action? What was it, in other words, that finally brought down one misogynist television star—but elevated another to the presidency?

* * *

For mainstream opinion to crash against O’Reilly and Ailes, the public needed to see hard evidence of lawsuits from—and settlements to—more than one of their alleged victims. Once that happened, in O’Reilly’s case, people pressed advertisers to abandon his show, and more than 50 of the most prominent companies did. There was no big advertiser boycott against Ailes, probably because few companies would be able to justify pulling ads from an entire very popular network. Instead, Ailes was done in by the status of the women he victimized. Carlson was joined by Megyn Kelly and other big-name hosts in her accusations against Ailes; 21st Century Fox evidently came to realize that the outrage over the allegations wasn’t going to die down anytime soon and would cost them more money than Ailes was worth. The fact that recognizable personalities were putting their names on the line against Ailes probably convinced some people prone to distrust women that these allegations were true.

It’s worth noting that the decision-makers who finally ousted O’Reilly and Ailes were apparently not significantly bothered by the distressed women in their organization or by the men’s behavior itself, since they repeatedly declined to sanction it. And 21st Century Fox wasn’t being held accountable by Fox News’ viewers, who are accustomed to the network’s casual misogyny and objectification of women. (On Wednesday, while the rest of U.S. media outlets were reporting on O’Reilly’s departure, a Fox News host on The Five told his female co-host on air that her tight dress was giving America an erection. With O’Reilly’s departure, that show is slated to move to the prime 9 p.m. timeslot.)

O’Reilly’s ratings rose after his past sexual harassment settlements blew up in early April. No, the Fox leaders that pushed out O’Reilly were responding to everyone else: people who don’t watch Fox News but do patronize the businesses that advertise on it, and so pursued those companies to abandon the show. The Murdoch family, which runs 21st Century Fox, only cut ties with O’Reilly after an internal investigation uncovered further complaints against him. They may also have worried that an impression of a lax sexual-harassment policy at Fox News would endanger their current bid to take over European broadcaster Sky.

Unlike Ailes and O’Reilly, Cosby wasn’t a marquee name by the time he reached the limit of his public goodwill. Sure, reruns of The Cosby Show were still airing, and TV Land pulled them. Netflix canceled plans for a comedy special. NBC scrapped a comedy show, still in its very early stages of development, that would have featured him, and he resigned from the board of trustees at Temple University. These ramifications came within a few weeks of Buress’ reminder that Cosby had been accused of sexual assault by 14 women.

But Cosby was already old, rich, and semi-retired, which limited the consequences he suffered from rising public condemnation. He finished a wildly successful comedy tour, during the span of which multiple new women spoke up with details of his abuse. Thanks to his personal and financial relationship with the National Museum of African Art in Washington, a scheduled exhibition of his and his wife’s personal art collection proceeded as planned, full of flattering depictions of the comedian’s family. It took a unified denunciation from some of Cosby’s alleged victims for the National Museum of African American History and Culture to insert a mention of his alleged crimes in a permanent display about his career.

Cosby has also been significantly protected by the justice system. Statutes of limitations have prevented most of his alleged victims from holding him accountable in court, even as he’s filed defamation suits against them for harming his reputation. The first set of criminal charges didn’t come down against Cosby until late 2015, when prosecutors first got access to transcripts of Cosby admitting he’d used quaaludes to coerce women into sex. A federal judge finally unsealed that evidence, obtained in a deposition from a 2005 civil case against Cosby, just before the 12-year statute of limitations was about to expire, and jury selection for his trial is set to begin in May. The outrage that followed public allegations of dozens of unpunished crimes seems to have pressed the justice system to do its work.

Cosby may have been shamed, but his wealth buys him privacy while his lawyers drag out the case against him. If he’s convicted, all his alleged victims but one will still be “alleged.” Ailes and O’Reilly will do just fine, too, with the reported $40 million and $25 million they respectively received when they left Fox News. (Gretchen Carlson, for comparison, got a $20 million settlement. O’Reilly’s accusers got a combined $13 million.) In the days after allegations against them provoked public furor, the president of the United States backed and praised both men, and Ailes ended up as Trump’s adviser.

This brings us to the remarkable evasion of accountability achieved by the man who must be the most famous alleged sexual abuser in the world: President Trump. In the run-up to the election, about a dozen women gave public accounts of how he had allegedly harassed and assaulted them, risking their reputations as Trump responded with insults and smears. Though Trump reaped enormous publicity and cable-news airtime for his outrageous statements, sensational rallies, and pure swagger, nearly every U.S. newspaper—including some that had never made endorsements before—ultimately advised readers to vote against him. Journalists covered the sexual assault allegations in depth.

Some voters might have been predisposed to dismiss these women’s allegations as exaggerations or lies. But the Access Hollywood video released before the majority of Trump’s accusers went public proved that, in private, Trump was telling precisely the same story. In that unbelievable moment, I was sure the GOP would drop him. Why would a party already struggling with female voters continue to support a man who bragged about abusing them? Over the weekend that followed the Friday revelation of the tape, Republican leaders began to withdraw their support.

But Trump refused to be shamed—and Republicans gradually returned to the fold. Instead of advertisers that people of all political stripes could boycott, Trump’s allies included Republican politicians who saw they could benefit from having a suggestible egomaniac in the White House. Trump’s base, and the constituents of those Republican legislators, were leery of a female candidate (a Clinton, no less) and perhaps more tolerant than average of a rich white man who abused women. Polls have shown that registered Republicans are far less likely than Democrats to consider workplace sexual harassment “a problem.”

Still, plenty of those constituents must have been appalled by the allegations against Trump. Where was their outcry? Faced with an alternative they hated, and no clear method so close to the election to agree on a more palatable candidate—Mike Pence, even!—many brushed off the allegations and decided to move forward with their man. Some of the members of Congress who’d leaped off the bandwagon clambered back on. Meanwhile, three-quarters of eligible American voters chose Hillary Clinton, other candidates, or no candidate at all.

But, in the end, 1 in 4 were willing to accept an admitted sexual predator as president, and that was enough. As Trump would probably remind me, the election is over, and he won. Perhaps there wasn’t as much disgust and anger over his alleged crimes as it seemed. Perhaps there wasn’t enough time for supporters to alter their course or simply too many other countervailing factors: the Comey letter, Russian interference, sheer sexism against Hillary, the damn emails. Certainly, there were no advertisers or courts for the majority of Americans to intervene with, not about this. And there was no secret, brilliant plan to stop him. There were only people.

November seems like a long time ago now—a pre–Women’s March, pre-“resistance,” pre–Obamacare town hall, pre–post-Trump world. Today, would things go differently? You might look at the fall of Bill O’Reilly and wonder whether the public has learned some things since Trump’s election about getting sexual predators out of power.

Here’s the problem with that argument: We’ve heard it before. Some argued that women were newly emboldened to push back on assault in 2015 when Cosby’s accusers finally flipped his reputation—at the time, New York’s Noreen Malone wrote that “more has changed in the past few years for women who allege rape than in all the decades since the women’s movement began.” When Ailes was booted from Fox News, my Slate colleague Nora Caplan-Bricker wrote that his ouster came in a “post–Bill Cosby era” in which people are less likely to make excuses for high-profile men who abuse women. A few months later, when several women accused Trump of sexual assault, it seemed like the combination of the Ailes scandal, the downfall of Cosby, and the surge of college activism around sexual assault (remember Brock Turner?) would have been enough to stop an Ailes clone from becoming president. It didn’t. If our supposed mass awakening to the credible trauma of sexual assault victims couldn’t keep Trump from the Oval Office, how woke are we, anyway?

The Washington Post’s Margaret Sullivan credits Carlson, who filed the first suit against Ailes, with starting “an avalanche” that brought down “two of the most powerful men in media.” “The world is suddenly a different place for women who’ve experienced sexual harassment in their workplaces,” Sullivan writes. Now, “they may find it easier to tell their stories. Easier to be brave.”

That line brought to my mind a Sara Bareilles song, the one that goes, “Say what you wanna say / And let the words fall out / Honestly, I wanna see you be brave”—a song adopted by the Hillary Clinton campaign. Her campaign failed in spite of the bravery of Summer Zervos, Natasha Stoynoff, and Kristin Anderson, who spoke up about their disturbing, often frightening experiences of assault by a wealthy real estate developer who was, by then, leading the Republican Party.

Without enough people rallying behind them, without the levers of advertisers or a media conglomerate worried about looking civilized enough to make new deals, this kind of bravery was not enough. The GOP establishment and far too many voters found it easy enough to shrug and choose Trump anyway. In 2017, it is more possible than ever for women to speak up about assault and harassment, but that’s only half the battle. People still have to decide to listen.