(CNN) For years, John Christopher Smith suffered in silence as his manager at a South Carolina restaurant physically abused him and forced him to work over 100 hours a week without pay, federal officials say.

Bobby Paul Edwards, 53, used violence, threats and intimidation on the African-American employee who worked at the eatery in Conway, according to the US Justice Department. It said Edwards hurled racial epithets at him, beat him with a belt, and hit him with pots and pans.

This week, Edwards pleaded guilty to one count of forced labor in the incidents, which allegedly happened when he managed the restaurant between 2009 and 2014.

Authorities said Smith had intellectual disabilities, and was afraid of losing his job. At times, Edwards, who's white, would burn Smith's bare neck with hot tongs to make him work faster, federal authorities said. Other times, he'd allegedly beat him with a belt and punch him with his fists.

"Human trafficking through forced labor can happen on farms, in homes, and as today's case shows -- in public places, such as restaurants," Acting Assistant Attorney General John Gore said Tuesday.

Read More