Washington (CNN) Former Republican Rep. Joe Walsh is calling for a primary challenger to take on President Donald Trump in 2020, as he is reckoning with his past controversial comments and apologizing for his role in helping elect an "unfit con man" to the presidency.

"If Republicans don't stand up right now and challenge this guy right now, he's bad for the party. He's bad for the country. We're going to get wiped out in 2020," Walsh said Thursday in an interview on CNN's "New Day," echoing his recent op-ed in the New York Times

Walsh, now a conservative radio host, said he voted for Trump in 2016, but only because he wasn't Hillary Clinton. Where Trump lost him, Walsh said, was during his press conference with Vladimir Putin in Helsinki last year in which he sided with the Russian strongman over his own intelligence community's assessments of interference in the 2016 election.

In his Times op-ed published Wednesday, Walsh argues that Trump -- who enjoys high approval ratings among Republicans -- is "more vulnerable to a challenge from the right" rather than a centrist challenge, pointing to former Massachusetts Gov. Bill Weld's GOP candidacy.

"I'm on the right, and I'm hugely disappointed that challenge hasn't yet materialized," Walsh writes.

Walsh told CNN's John Berman on Thursday that he knows "a lot of good people are thinking about primarying Trump," without naming names.

"It's got to be the moral case. This has got to be a mission. This guy is unfit to be President," Walsh said.

He argued that what Republicans need is "somebody to stick their neck out and say, 'Enough. Enough, President Trump. We're tired of the lies. We're tired of the bullying. We're tired of the cruelty. Enough.' Somebody has to break that impasse."

In his op-ed, Walsh accuses Trump of being a "racial arsonist who encourages bigotry and xenophobia to rouse his base and advance his electoral prospects."

"One of the reasons I wrote this op-ed was to apologize -- to apologize for the role that I played in putting an unfit con man in the White House," Walsh said on CNN Thursday.

"The country was divided before Trump that's why we got Trump. I was at the head of that divide talking about and pushing ideas I believed in," Walsh added. "I was pushing ideas that I believed in. I've been very outspoken. Oftentimes, I stepped over the line."

Walsh said the election of Trump has made him realize "how ugly our side -- both sides can get -- but how ugly I've been. It's caused me to reflect, and no longer engage in personal attacks and just focus on the policy differences."