Story highlights Paul Ryan is facing heat for his Donald Trump stance, Rep. Mark Meadows says

The House speaker has had a turbulent relationship to his party's presidential nominee

(CNN) The conservative Republican congressman who led the charge to oust former Speaker John Boehner is now agitating for the removal of his replacement.

Rep. Mark Meadows said Thursday the effort to remove Speaker Paul Ryan is "picking up some steam" because many GOP lawmakers and a stream of callers to the North Carolinian's congressional offices are incensed the Wisconsin Republican hasn't embraced fully Donald Trump's candidacy for president.

"A lot of people who believe so desperately that we need to put Donald Trump in the White House -- they question the loyalty of the speaker," Meadows told radio host Tyler Cralle of WAAV radio in Wilmington, North Carolina.

Meadows, one of the 30-plus members of the ultra-conservative and powerful House Freedom Caucus, said there "will be real discussions after November 8 on who our leadership will be and what that will look like going forward."

House Republicans are expected to meet privately the week after the election to vote for their leaders.