Olmsted called them “fat, black, and stupid,” and said things like “You will never amount to anything,” and “You only have one parent,” according to a lawsuit filed by three families. He also allegedly forced black students to sit in the back of his classroom, facing the wall, without books.

In their suit, the parents said they’d made some 100 complaints to Heights’ principal Jayne Ropella over a three-month period. They cited Olmsted’s professional file with St. Paul Public Schools — which showed that he once gave a sixth-grader a sexually suggestive birthday card and got stupid handsy with a co-worker at a holiday party — as evidence that he shouldn’t be allowed to work with kids.

Other teachers who took issue with Olmsted echoed their frantic calls for an investigation. Finally, in January 2012, Olmsted was suspended and put on paid leave. He resigned eight days after that, and the parents’ lawsuit was settled a year later.

Olmsted came to regret his resignation, however.

Three months later, Olmsted tried to renege. He wanted to go back to teaching driver’s ed, he said. When St. Paul refused to take him back, Olmsted sued. He alleged that he was forced to resign under duress, that he wasn’t thinking clearly.

Olmsted claimed the district lawyer suggested he that resign quietly, or the district would issue a report to the Board of Teaching describing all the bad shit that its internal investigation uncovered. The district didn’t say what specific termination charges it wanted to pursue. Olmsted said he felt like the district was placing a “gun to [his] head.”

The district court ruled that Olmsted couldn’t force St. Paul Public Schools to accept the revocation of his resignation. Olmsted decided to appeal.

On Monday, the U.S. appeals court agreed with the lower court. There’s no evidence to support the claim that the district bullied him into resigning, according to the ruling. Instead, Olmsted should have known that he was on the way out, either through termination or resignation, “pending further investigation of allegations of serious misconduct.”

Besides, Olmsted had about eight days to mull over his options — which were all laid out for him by a union attorney — before he submitted his unprompted resignation. That should have been enough time to make up his mind, the court concluded.