Larry Barton, 75, was ousted by voters in Talladega on Tuesday as he sought a fifth term

Maybe not all publicity is good publicity.

Voters in Talladega, Alabama, rejected Mayor Larry Barton’s bid for a fifth term on Tuesday, two weeks after he was assaulted Aug. 8 by a man whose ex-attorney says Barton was caught on tape having sex with the assailant’s wife.

Get push notifications with news, features and more.

As voters headed for the polls early Tuesday, a confident Barton told AL.com: “We didn’t have to do much publicity. Of course, I don’t like this kind of publicity, but all publicity of a sort is good.”

After the results were in, giving the victory to Jerry Cooper Sr. by a nearly two-to-one margin, Barton updated his comments to the website. “I have been honored to serve four terms, and there’s only two mayors of Talladega who have served that long,” he said. “I count it as an honor. I’ve had four good terms.”

Before Tuesday, Barton told PEOPLE he thought the sex tape allegation was “possibly politically motivated.”

That allegation – made by Stewart Springer, a former attorney for Barton’s accused assailant, Benny Green, 71 – named Barton, 75, as the other man on a video that showed Green’s wife, Charlotte, 68, engaged in sex acts in the back of the Greens’ Talladega liquor store with a man who was not her husband. The tapes, allegedly documenting three separate incidents in December 2013, are described in court papers that are part of Benny Green’s pending request for a divorce.

Neither Charlotte Green nor her attorney, Steven Adcock, returned repeated calls by PEOPLE.

Benny Green is charged with first-degree assault in the attack on Barton. Police said a suspect wearing a trench coat and wig approached Barton outside of the barber shop where he cuts hair part-time, then hit Barton with a baseball bat before attempting to flee by bicycle. The injuries briefly put Barton in the hospital.

Green has not yet entered a plea. The charges carry a penalty of two to 20 years in prison. The onetime friends previously had been co-hosts of Barton’s cable TV show, In the Interest of the People.

Tuesday’s loss was not the first mayoral defeat for Barton. He was a three-term mayor when he was removed from office in 1994 following his conviction on fraud and money-laundering charges tied to a $5,900 theft from the city. After serving three years in prison, he ran again and was defeated again before winning a fourth term in 2011.