Ever since Ashley Judd and others came forward to say that Harvey Weinstein had harassed or assaulted them, the list of powerful men accused of similar behavior has grown longer by the day. Leon Wieseltier. James Toback. Mark Halperin. Brett Ratner. Kevin Spacey. Michael Oreskes. Ed Westwick. Louis C.K.

At this point, the most important question to ask is not who’s next, but what.

Some men have issued apologies in which they admit to some (though generally not all) of the allegations against them. Some have been removed from their positions, their power diminished if not destroyed.

Those who have come forward publicly have helped others — many women, and some men, now feel safe speaking out after years of silence. Some who were brushed off or disbelieved have been vindicated. Some people no longer have to work alongside their harassers. These advances are real and valuable.

It’s not yet clear, however, whether the swell of public testimony and the growing list of firings will lead to lasting change across industries. Will the reckoning stop when the news cycle moves on? Will CEOs congratulate themselves on getting rid of the bad apples and call the problem solved? Or will we look deeper, at the inequalities that keep harassers safe and victims silent and that will take more than a few firings to undo?

One inauspicious sign is the fact that some people are already asking whether the “pendulum” is “swinging too far.” The implication of such questions is that while sexual assault is clearly beyond the pale, we should be careful about over-punishing men for lesser offenses. Writer and editor Cathy Young expressed a similar point of view when she told NPR she was "concerned about this mindset that we have to constantly police for microaggressions — which, a lot of that is defined very subjectively."

It’s certainly true that since women’s accounts of their experiences with Weinstein were first published, other women have reported a range of experiences from inappropriate behavior to sexual assault, both publicly and in forums intended to be private, like the Shitty Media Men list. The appropriate consequences, professional and legal, for someone who makes an unsolicited comment about a subordinate’s appearance will, of course, be different than for someone who commits rape. But we’re taking too narrow a view of the problem if we assume that all that matters is consequences for men.

Fighting sexual harassment means helping women, not just punishing men

To really address the problem of sexual harassment in the workplace, we need to address its affects on women. In a recent Washington Post/ABC News poll, a third of women said they had experienced sexual advances from a male colleague or a man with influence over their job; a third of those women said the advances constituted sexual abuse. The majority of people who experience harassment don’t report it, and of those who do come forward, 75 percent face some kind of retaliation, according to a 2003 study.

The consequences of experiencing harassment can be severe. Over and over in recent weeks, women have described not just feeling unsafe or uncomfortable with a man, but feeling that their careers were compromised as a result, and that they had nowhere to turn.

Fixing this problem isn’t just about punishing men. It’s about making sure women have ways to report grievances that go beyond vague directives to talk to their managers (who may have harassed them) or file a complaint with HR (which is sometimes more worried about protecting the company than helping employees). It’s about giving women access to the career opportunities men enjoy, so they’re not forced to avoid predators while men benefit from their mentorship. Ultimately, it’s about making women equal participants in the workplace.

Right now, a lot of people are asking how to deal with men who violate appropriate boundaries. This is an important question, but it’s far from sufficient. We also need to ask what we can do to help women feel safe at work, how we can make sure they have the same chance to succeed that men have. Real change won’t happen until we can answer those questions.

As we’ve seen from men’s reports about allegedly being harassed or assaulted by Kevin Spacey, women aren’t the only ones who can be targeted, and power imbalances can put people at risk even when gender inequality isn’t a factor. But it remains true that for many women and in many workplaces, gender inequality is a huge factor, one that has to be addressed in order to truly address harassment. As long as women are second-class citizens at work, they will never be safe from predatory men.

As Margaret Sullivan put it recently at the Washington Post, “a few powerful men have been shamed or demoted, but the underlying issues of gender inequality and power dynamics live on.” In media, she noted, “having a critical mass of female decision-makers, rather than a token presence, allows ideas to bubble up and voices to be heard in new ways. This is, of course, true for racial diversity, too.”

Women’s representation in media has been improving, but slowly. Women made up 39.1 percent of newsroom employees in 2017, up from 38.7 percent in 2016 and 37.35 percent in 2001, according to a survey by the American Society of News Editors. Among newsroom leaders (which publications were allowed to define for themselves), 38.9 percent were female in 2017, compared with 37.1 percent in 2016. At all S&P 500 companies, just 26.5 percent of senior and executive level managers, and 5.2 percent of CEOs, are female, according to Catalyst. Only 4.7 percent of executive and senior managers are women of color.

Representation is critical for preventing sexual harassment and creating an environment in which women can thrive. It’s also critical that the conversation about harassment extend beyond the media and Hollywood, industries that tend to capture headlines. Restaurant workers are especially vulnerable to harassment and may have little recourse; domestic workers have historically lacked the legal protections against harassment that some other workers have. Sexual harassment is a labor issue that is only beginning to be treated as such.

What comes after Weinstein, then — and after Toback, Oreskes, C.K., and all the rest — has to be about helping all women, not just disciplining a few men. The goal can’t simply be a workplace where harassment is not tolerated — after all, most companies already make this claim — but one where workers are valued and treated equally. If we think about the issue this way, concerns about the pendulum swinging too far start to make less sense. After all, are we worried that women will be too safe?