A Republican congressman has said he would have beaten up Roy Moore, the GOP nominee for a US Senate seat in Alabama, if any of the women who have made sexual misconduct allegations against the judge had been his daughter.

“The 14-year-old girl that was there, I can tell you right now if it was my daughter, I’d break his face, I’d break his fingers, and I’d probably do a lot worse,” Virginia representative Scott Taylor told CNN on Wednesday.

Taylor, a former navy Seal, said he did not “feel comfortable” with Moore’s response to the sexual misconduct allegations against him, which include the claim that he initiated a sexual encounter with a 14-year-old girl in 1979, when he was 32. Moore has denied all such allegations.

“All I know is what I’ve seen,” Taylor said. “I saw the man give his interview. Me personally, I don’t think it was sufficient enough.”

Moore’s campaign – which on Wednesday confirmed the departure of its communications director, John Rogers – did not immediately return a request for comment.

The Republican party is increasingly at odds with Donald Trump over Moore and the Alabama race. On Tuesday, the president indicated that he did not accept the accusations against Moore. The Senate majority leader, Mitch McConnell, and other senior Republicans have said they do.

Asked how he felt about the potential election of an alleged “child molester” to the Senate, the president said Moore “totally denies it”.

“He says it didn’t happen,” Trump told reporters. “You have to listen to him, also.”

Republican leaders are concerned that win or lose in Alabama on 12 December, Moore will damage the party’s brand before the 2018 midterm elections.

As Moore has refused to drop out, controversy over alleged sexual misconduct by prominent men has spread in politics to include accusations against Trump, Bill Clinton and two congressional Democrats, the Michigan representative John Conyers and Minnesota senator Al Franken.

The Washington Post published its explosive allegations against Moore two weeks ago. Since then, other women have come forward. Moore, near-level with Democrat Doug Jones in the polls, has said he does not recall dating any young women without their mothers’ permission.

“If you are a liberal and hate Judge Moore, apparently he groped you,” his campaign told the Post last week. “If you are a conservative and love Judge Moore, you know these allegations are a political farce.”

In his remarks this week, Trump came close to endorsing Moore, saying: “We don’t need a liberal person ... a Democrat” in the seat vacated by attorney general Jeff Sessions.

Speaking to CNN, Taylor suggested that though Trump supported Moore’s opponent, Luther Strange, in the primary, the president had “probably looked at raw politics and the alternative, of course, would jeopardize his agenda in a very close Senate”, where tax cut legislation is pending.

The congressman added: “I think you have to listen to the women. Clearly, this isn’t an isolated case now.”

Taylor represents Virginia’s second congressional district, a competitive seat in a state that saw Democratic successes in elections this month.