Rep. Justin Amash (R-MI), a member of the conservative House Freedom Caucus, on Monday said that House Speaker Paul Ryan (R-WI) either needs to make changes or step down from his role leading the Republican caucus.

“We need either a change in direction from this speaker, or we need a new speaker,” Amash said at a town hall in Michigan, according to CNN.

Members of the Freedom Caucus helped oust former House Speaker John Boehner (R-OH), and are known for their hard-line stances. They opposed the House leadership’s bill to repeal and replace Obamacare, contributing to the legislation’s failure. However, leadership’s inability to whip enough support for the bill’s passage has not led to a flood of calls for Ryan to step aside. The chair of the House Freedom Caucus, Rep. Mark Meadows (R-NC), said at the end of March that there had not been discussions about replacing Ryan as speaker.

Amash also criticized President Donald Trump and Ryan for giving Republicans talking points to use during the House recess.

“When we go home for the weekend, they give us a set of talking points. They say ‘here are your talking points,'” he said at the town hall, per CNN. “That’s not the way you’re supposed to represent a community.”