Sen. Mitt Romney is not ready to endorse his party's incumbent president for re-election.

The Utah Republican said Sunday on CNN "State of the Union" that it's "way too early" to declare his support for President Donald Trump in 2020, saying Trump needs to improve "when it comes to helping shape the character of the country."

Polling shows that the president is heavily favored to defeat former Massachusetts Gov. Bill Weld, his only primary challenger to date. But even if Trump secures the nomination, Romney may cast his vote for the same woman as he did in 2016: his wife, Ann Romney.

"We will have to wait and see how she does," Romney quipped.

Romney was one of Trump's fiercest Republican critics during the 2016 campaign, calling him a "phony" and a "fraud." And he has remained critical since Trump took office, writing in a Washington Post op-ed that "the president has not risen to the mantle of the office."

Trump responded by calling on Romney to be a "team player" and taunting him about his 2012 loss to President Barack Obama, saying he could have won if had "fought the way he fights me."

"I think young people, as well as people around the world, look at the president of the United States and say, does he exhibit the kind of qualities that we would want to emulate?" Romney said.

"And those are qualities of humility, of honesty, integrity," he said. And there "the president has distanced himself from some of the best qualities of the human character."

But Romney does not believe that impeachment is the right way to go, he said Sunday. Although he found Rep. Justin Amash "courageous" for becoming thefirst congressional Republican to say that Trump was guilty of "impeachable conduct, he said Amash "reached a different conclusion than I have."

More:Trump attacks Rep. Amash as a 'loser' and 'lightweight' after the Republican calls for impeachment

Romney said that after reading special counsel Robert Mueller's report on Russian election interference on potential obstruction of justice by the president, "you just don't have the elements" to prove that a crime was committed.

He acknowledged that impeachment is as much about politics as the law, but he said there is not enough support for it among the American people.

"And I think those that are considering impeachment have to look also at the jury, which would be the Senate. The Senate is certainly not there either," he added.