Amid the latest wave of sexual harassment allegations, you might conclude, with some relief, that predators are finally being held accountable for long-repressed abuse. But while several men have lost their jobs, there’s been little accountability in a legal sense. Several cases reported in the media—from Harvey Weinstein to Kevin Spacey to Russell Simmons—would constitute crimes if proven, but thus far there haven’t been any indictments. In Weinstein’s case there have been reports of criminal investigations, but in general the #MeToo movement has played out in the press and on social media.

This makes many uncomfortable. Each story has its own particulars, but they all inspire demands for the same conclusion: effectively, banishment from the public square. Bill Maher warned against lumping in Al Franken’s alleged groping with Roy Moore’s alleged stalking of minors. The mantra to “believe women” bumps up against questionable accusations, such as assault allegations against Senator Richard Blumenthal that were made by what appears to be a Twitter bot. Social media doesn’t make allowances for legal concepts like the presumption of innocence, and it can justifiably lead to fears of mob rule or partisan exploitation.

But we should identify the real culprit for this state of affairs: the long, slow abandonment of the rule of law in America. The reason adjudicating sexual misconduct claims has been left to the media and the crowd is that people have no expectation that the legal system will adjudicate those claims fairly. How can anyone blame them? They have witnessed endless instances of powerful people, mostly wealthy men, getting away with criminality and deception, in every context imaginable. When you don’t have a working justice system, you get a kind of vigilantism as a result. The problem isn’t the vigilantism—it’s the broken framework that leads desperate people to take matters into their own hands.

That powerful people face little sanction for misbehavior is an old story, as true in gender as it is in class. But brazen impunity for the powerful is a hallmark of our era. The worst financial crisis in America in nearly a century led to practically no convictions for those whose actions facilitated the meltdown. The Catholic Church shuttled around sex-abusing priests for decades with little reckoning. Cops shoot black people and go back on the job.

On the opposite end of the spectrum we have a wildly punitive justice system for those unlucky enough to have no money or power. Black men are sentenced to more jail time for committing the exact same crimes as whites. Nonviolent drug offenders rot in jail. Mandatory minimum sentences and three strikes laws trap many in an unforgiving correctional system.