A black judge in Mississippi claims he was removed from office following years of resistance from a mostly white legal system to his push to treat drug offenders instead of imprisoning them.

Justice Court Judge Rickey Thompson of Lee County—named for Confederate general Robert E. Lee—is the first and only black judge in the county’s 149-year history. In May, the Mississippi Supreme Court removed him from the post that he’s held since 2004 over a slew of misconduct allegations.

“It came to the point where they couldn’t beat me at the ballot, so they had to find another direction,” he said.

Thompson said the sheriff quit sending him suspects and fellow judges quit sending the accused to his drug court. Warrants he signed for the sheriff’s department went unserved. A bailiff even once refused to open his court.

Officially, Thompson was found by Mississippi’s highest court to have violated several rules, including what amounts to making clerical errors like using the wrong form when filing paperwork. That should not be surprising given Mississippi requires only a high school diploma to be a justice court judge. (Thompson, a registered nurse with an associate degree, has more education than that of his fellow justice court magistrates.)

Thompson’s crimes include speaking up for a bail bondsman who had been previously suspended from operating by the sheriff; preventing a drug court defendant from choosing her own attorney over one he advocated for; keeping some people in drug court longer than the two years allowed by state law; wrongly incarcerating four people.

The last charge is the most important: When a handful of offenders violated the drug court’s rules, Thompson found them in contempt of court and sent them to jail. The longest sentence was six months.

If Thompson hadn’t created his drug court in the first place, the four people he locked up for contempt would have almost surely been in prison longer than they would have been for their drug offenses. That means Thompson was kicked off the bench, in part, for not locking up drug offenders for longer than he did.

The judge was popular with blacks in the county, and he kept quite a few of them out of jail for minor offenses like marijuana possession. The sheriff evidently wanted to see them in his jail or prison.

“When I first started, obviously there was a learning curve, but you see some of the bad things that are going on,” Thompson said. “And as people got to know me and know that I was going to be fair, that I wasn’t going to be a rubber stamp, that’s when the trouble started.”

A complaint filed in federal court last week by Thompson’s attorney Jim Waide details what they say happened next.

In 2009, Thompson had seen enough people getting locked up for petty drug offenses and, after much negotiation and arm-twisting, convinced the county to start what would become his drug court. Thompson’s complaint alleges fellow judges—all white—began to refuse to send suspects—mostly black—to the drug court once it was running.

The resistance didn’t stop there.

When Thompson found a black misdemeanor suspect not guilty of disorderly conduct and resisting arrest, Sheriff Jim Johnson took revenge, Thompson claims.

Mississippi sheriffs decide which judges get which suspects. So when Thompson acquitted the black defendant, Sheriff Johnson simply quit bringing suspects to Thompson’s courtroom.

Johnson’s deputies refused to serve warrants issued by Thompson, he said. When Thompson held a contempt hearing for a white deputy, he said other deputies appeared in court to intimidate him. A bailiff once even refused to open Thompson’s court (the traditional “all rise”) while eight inmates sat in the courtroom without supervision, according to a letter from a counselor submitted as evidence in Thompson’s complaint.

In November 2013, the Mississippi Commission on Judicial Performance opened an investigation of Thompson’s alleged misdeeds and last summer recommended he be removed from the bench. The state’s supreme court agreed and removed him in May.

According to his attorney, the crimes for which Thompson was found guilty amount to offenses that are regularly committed by white judges who are not removed from office. The court said there was no evidence of racial bias.

So Thompson has sued in federal court the Mississippi attorney general, the local Democratic Party, and the county’s election commission to be reinstated and allowed to run for office.

Not content with waiting for the courts, Thompson ran for reelection and beat five competitors with 55 percent of the vote in the primary this month. It’s unknown whether Thompson will be allowed to run in the general election, though. Local Democrats and the county election commission cite a state statute that bars public officials who have been removed from office from running again. Thompson’s attorney says that is not in the state constitution.

After 11 years on the bench and three reelections, Thompson now has to fight the system he was once part of.