According to a recent New York Times article, we’re past due for a #MeToo-style national reckoning around the open secret of anti-mom bias in the workplace. In her op-ed, Katherine Goldstein lays out the ways in which working mothers are assumed to be less dedicated to their jobs and calls for a “national discussion about discrimination against mothers, one that begins to raise in the collective consciousness the notion that this kind of discrimination is wrong and truly harmful.” This discrimination can manifest in many ways, such as being passed over for a promotion to outright being fired. And as Goldstein notes, the consequences of this discrimination are huge. A working paper by the Census Bureau that Goldstein cites revealed that women who have children between 25 and 35 never close the pay gap between them and their husbands. Women who have children either before their careers have really started or after they’ve been well established go on to eventually close that gap.

But one of the more compelling bits of evidence for anti-mom bias didn’t come from the studies Goldstein cited, but from the backlash to her piece.

While many working mothers shared stories of discrimination online, a great deal of bitterness was targeted against those moms themselves. “I know this happens, but it’s not been my experience,” wrote one Twitter commentator. “I’ve always seen moms given a level a deference that’s [sic] causes problems for us childless folks.” One of the reader comments ranked by editors as a “Times Pick” was even more vitriolic: “I am one of the many women (a lawyer) without children who has had to work harder and longer to cover for pregnant women or mothers (also lawyers) whose first priority was always their family. Nothing is more irritating than women discussing diaper brands when they should be talking case law!” Another read, “You can call it bias, I call it ‘annoyance at having to pick up the slack for someone who cannot pull their weight because of a self-inflicted and avoidable condition.’ ”

The visceral anger and resentment evident in these comments shouldn’t be aimed at working mothers. The blame for both inflexible workplaces and anti-mom bias can be squarely rested upon the shoulders of an unfair and overly taxing work culture. As a colleague succinctly put it, “The problem is everyone in America works too hard and doesn’t take enough time off so we’re all bitter when we see anyone doing less or getting more than us.” Anti-mom resentment has more to do with the fact that most Americans don’t work in flexible work environments, so the only excuse for a “vacation” or “break” is major life event like pregnancy or a funeral. The resentment shouldn’t be targeted at people who take advantage of that sort of necessary time off—it should be targeted at the fact that the average worker doesn’t get enough general time off in the first place.

And that doesn’t even take into account the fact that most people who do jobs with a theoretical façade of flexibility—white-collar or creative work—rarely know how to set healthy boundaries in a world where workaholism is rewarded with promotions and Slack makes us available at all times. You shouldn’t need to be pregnant or have kids at home to get out of working late or answering emails at 9 p.m. While having kids does provide a graceful way to bow out of obligations that are unreasonable in the first place, childless workers should feel just as justified saying “No” to that project that requires extra hours.

Of course, a workplace culture where the power resides squarely with management makes it exceedingly difficult to set boundaries that don’t make you resent the woman who’s literally carrying a child. But that’s not her fault! That’s your manager’s fault and, more broadly, capitalism’s fault. So next time you find yourself feeling bitterness toward your working mom colleague, redirect that anger toward negotiating better work-life boundaries for everyone.