Story highlights Cantor added that voters "expect their leaders to live to a much higher standard"

He said that ultimately, "Alabama voters will decide"

Washington (CNN) Former GOP House Majority Leader Eric Cantor said Tuesday that Republican Senate candidate Roy Moore doesn't deserve to win the special election in Alabama.

"I think he deserves to lose," Cantor told CNN's "New Day." "And when I was in office as majority leader, I had the policy of zero tolerance for any of this kind of stuff."

Moore has been accused of pursuing sexual relationships with teenagers when he was in his 30s. In some cases, the women who have come forward about the relationships have alleged sexual abuse or assault. Moore has denied all the allegations.

Cantor added that voters "expect their leaders to live to a much higher standard, and that's what ought to be maintained."

President Donald Trump officially endorsed Moore on Monday, and the Republican National Committee has agreed to fund Moore's campaign after withdrawing support last month.

Read More