(CN) – Facing skepticism from a Sixth Circuit panel, a lawyer representing a former lawmaker expelled from the Tennessee House of Representatives for sexual harassment allegations argued he had standing to sue the state in federal court because it took away his pension and lifetime health insurance.

Judges wondered Friday what the point would be to remand the case back to the lower court and commented that some of former Tennessee lawmaker Jeremy Durham’s arguments were “really weak.”

In the Potter Stewart U.S. Courthouse in Cincinnati, Ohio, attorney Bill Harbison made what he said was a simple argument before U.S. Circuit Judges John K. Bush and John M. Rogers and U.S. District Judge Michael H. Watson, sitting by designation from the Southern District of Ohio. Harbison asked the judges to reverse what he said was an “error of law.”

“We have alleged that his pension and health insurance benefits were deprived because he was voted out of office unconstitutionally,” Harbison said.

But when Durham filed suit in federal court in order to recover those benefits, the judge there dismissed the suit for lack of standing, saying the lawmaker sued the wrong people.

While the Tennessee House made the vote to expel Durham, it was the state’s administrators that carried out the effects of that vote. Durham filed the suit against Larry Martin, commissioner of the Tennessee Division of Finance and Administration, Connie Ridley, the state’s director of legislative administration, and David Lillard Jr., Tennessee’s treasurer.

“We’re very simply asking that the court reverse the ruling based on standing and reverse the dismissal because we would suggest that the ruling in this instance does violence to the doctrine of ex-parte Young,” Harbison said, citing U.S. Supreme Court precedent.

In September 2016, the Tennessee House voted 70-2 to expel Durham after the Tennessee attorney general’s office issued a report that said the lawmaker sexually harassed women while in office.

It was the first time Tennessee lawmakers voted to expel one of their colleagues in 36 years.

Examples of the behavior, according to the report, included giving a college student who was not yet 21 alcohol, “followed by sexual intercourse.” He also allegedly made suggestive comments to a female lobbyist, “that he expected something in return for supporting her bill.”

Durham had already lost the primary in his home district before the ouster.

According to the suit, Governor Bill Haslam convened a special session of the Legislature to address federal highway funds, during which Durham was ousted. Durham’s complaint said the Tennessee Constitution specifies a special session can only address the issue for which they were convened.

But Representative Susan Lynn, R-Mt. Juliet, introduced the motion that eventually expelled Durham from the legislative body.

Durham had two months left before his term ended.

In an email, the Tennessee Benefits Administration told Durham that because he was expelled, he was not eligible for benefits.

Judges on the Sixth Circuit panel wondered if the case was not a due process claim, as the lawmaker alleged, but rather a disagreement over the interpretation of Tennessee law, specifically the state’s constitution. If so, the state would be immune in federal court.

As for her part, Deputy Attorney General Janet Kleinfelter said the administrators named as defendants in Durham’s suit merely applied the law – just as they would to any other state employee.

“There are not allegations in the complaint that says these statutes are unconstitutional,” Kleinfelter said. “There’s not allegation in the complaint that these defendants improperly interpreted the statutes or improperly applied them, or acted beyond the scope of their authority.”

It is unclear when the court will issue a ruling in Durham’s case.