When it comes to misogyny and white supremacy, we’ve held the wrong audience accountable.

For years, Harvey Weinstein’s and Donald Trump’s private audiences could be divided into two types of (often) men: his vocal supporters and his silent endorsers.

The outspoken supporters — whether casual misogynists or white supremacists — are henchmen who helped take down women’s careers or allies in Congress who are themselves proponents of a white nationalist agenda. Most critics of both Weinstein and Trump consider this “base” group the real problem. For they are the fringe who have somehow become the norm.

But the people who really sustain misogyny and white supremacy are the quiet endorsers. Because they have the power to denounce the words, to reject the assumptions and, if necessary, to walk out of the room. They were the first to know and could have rallied — or become — the opposition.

And because so many of us have acted like quiet endorsers, Weinstein and Trump have the power they have (or, in Weinstein’s case, had).

Fortunately, the Weinsteins of our world are finally losing their power. This is in part because their actions behind their closed doors have been outed by brave women. But they could not have built that power in the first place without the presence of men (and some women) who failed to ask questions, or looked the other way or ignored their own beliefs about the equal value of women. #MeToo will become a movement of transformative change when brothers join their sisters to say #TimesUp.

White supremacy deserves the same fate. Trump has enjoyed a coterie of supporters and endorsers of white supremacy. Putting his misogyny aside for a moment, much of Trump’s electoral base, Congressional allies and conservative media henchmen and women seemed to support his barely veiled animus toward people of Latin American or African descent and Muslims. Yet more worrying are those congressmen who were in the room when Trump called El Salvador, Haiti and the continent of Africa “shithole countries,” yet refused to speak out. They equivocated or offered “no comment,” quietly endorsing Trump’s bigotry.