"I'm just not ready to do that at this point. I'm not there right now," the Wisconsin Republican told CNN's "The Lead with Jake Tapper" in an interview.

Ryan's position makes him the highest-level GOP official to reject Trump since the real estate mogul became the last candidate standing in the party's nominating contest. His move gives down-ballot Republicans cover to hold off on supporting Trump. It could also keep his agenda in the House from being overtaken by Trump's policy positions.

Ryan said he hopes to eventually back Trump and "to be a part of this unifying process." The first moves, though, must come from Trump, he said.

Ryan said he wants Trump to unify "all wings of the Republican Party and the conservative movement" and then run a campaign that will allow Americans to "have something that they're proud to support and proud to be a part of."

"And we've got a ways to go from here to there," Ryan said.

Asked whether Trump's proposed Muslim ban, his opposition to free trade and his call to deport 12 million undocumented immigrants would preclude him from ever supporting Trump, Ryan said: "We got work to do."

Trump responded to Ryan late Thursday afternoon, saying in a statement that "I am not ready to support Speaker Ryan's agenda. Perhaps in the future we can work together and come to an agreement about what is best for the American people. They have been treated so badly for so long that it is about time for politicians to put them first!"

Trump didn't address Ryan's comments at a campaign event Thursday night in Charleston, West Virginia. But speaking to reporters at the event, Trump's campaign manager, Corey Lewandowski, dismissed the idea that the remarks are a slap in the face to the presumptive nominee coming from the leader of the Republican Party.

Lewandowski then pointed to onstage to Trump.

"That's the leader of the Republican Party," Lewandowski said, adding that it's too early to say where Trump and Ryan might find common ground in terms of their political agenda.

On Friday, Ryan's office announced that he, Trump and top Republicans will meet next Thursday.

"Having both said we need to unify the party, Speaker Ryan has invited Donald Trump to meet with members of the House Republican leadership in Washington on Thursday morning to begin a discussion about the kind of Republican principles and ideas that can win the support of the American people this November," the statement read. "The Speaker and Mr. Trump will also meet separately, along with (Republican National Committee) Chairman Reince Priebus."

Earlier Thursday, CNN reached out to 16 Republican elected officials, leaders and major fundraisers associated with former President Ronald Reagan, both Presidents Bush and former Vice President Dick Cheney. Speaking on background, the Republicans, like Ryan, said they were not yet ready to support Trump. Because of this, they said they were not planning to go to the convention, though they won't vote for Clinton.

After Ryan's announcement, however, several said they were thrilled at the House speaker's position.

Striking comments from Ryan

Ryan's comments were striking because Senate Majority Leader Mitch McConnell said Wednesday night that he'd back Trump.

Neither of the last two Republican presidents -- George H.W. Bush and George W. Bush -- will attend the GOP convention in Cleveland . Nor will the 2008 nominee, John McCain, or the 2012 nominee, Mitt Romney.

The House speaker said he'd only started considering whether he'd support Trump after the real estate mogul won Indiana's primary Tuesday -- knocking both Texas Sen. Ted Cruz and Ohio Gov. John Kasich out of the race and ending the possibility of a contested convention.

"I thought about this two days ago. I thought, actually, this thing was going to go to June 7 at the very least -- probably to a convention -- and so this is all pretty new for us," he said.

"The bulk of the burden on unifying the party will have to come from our presumptive nominee," Ryan said. "I don't want to underplay what he accomplished. ... But he also inherits something very special, that's very special to a lot of us. This is the party of Lincoln and Reagan and Jack Kemp. And we don't always nominate a Lincoln or a Reagan every four years, but we hope that our nominee aspires to be Lincoln- or Reagan-esque -- that that person advances the principles of our party and appeals to a wide, vast majority of Americans."

He continued: "And so, I think what is necessary to make this work, for this to unify, is to actually take our principles and advance them. And that's what we want to see. Saying we're unified doesn't in and of itself unify us, but actually taking the principles that we all believe in, showing that there's a dedication to those, and running a principled campaign that Republicans can be proud about and that can actually appeal to a majority of Americans -- that, to me, is what it takes to unify this party."

Decision came 'very fast'

Ryan's decision to oppose Trump, at least for now, came "very fast," a source familiar with Ryan's thinking told CNN, adding that the speaker "truly was not prepared for this. He really did not expect Cruz to drop out. He was prepared for a contested convention. And when this happened, he just decided to go with his gut -- which was to hold off."

His comments about Trump quickly became a political football, with Hillary Clinton's presidential campaign highlighting the speaker as part of a "growing list of conservatives rebuking Trump" in an email.

A Republican strategist involved in Senate races told CNN that he's worried Ryan has set up a situation that will be difficult for him to eventually get out of.

"What are the conditions by which Ryan will ever endorse? I don't know how this ends," the strategist said. "What would make him get to a yes on Trump? I'm not sure what Trump can do, other than change his positions."

The strategist added, "It helps people by giving them cover. On the other hand, if you think of all the people who have already gone out of their way to endorse Trump, they're asking themselves, 'I have already jumped into the pool, where is Ryan?'"

A friend of Ryan's said Thursday's announcement was "totally Paul being torn by his own conscience. He couldn't just say, 'Everything I believe in doesn't matter.'"

Ryan's remarks provide some cover for reluctant Republicans, especially for those running for reelection in blue states, the friend added.

The friend said Ryan could eventually shift -- but "the ball is now in (Trump's) court."

"Let's see what he does with it," the friend said.

A frequent critic

Ryan has expressed misgivings about Trump's campaign for months.

When Trump proposed indefinitely banning Muslims from the United States in December, Ryan responded that such a move is "not who we are as a party" and in violation of the Constitution.

"This is not conservatism," he said then , adding, "Some of our best and biggest allies in this struggle and fight against radical Islam terror are Muslims."

In March, Ryan slammed Trump's refusal to disavow the support of former Ku Klux Klan leader David Duke in an interview with Tapper on CNN's "State of the Union."

"If a person wants to be the nominee of the Republican Party, there can be no evasion and no games. They must reject any group or cause that is built on bigotry. This party does not prey on people's prejudices," Ryan told reporters on Capitol Hill

Later that month, he called the violent skirmishes that had broken out at Trump's rallies " very concerning " and said that candidates must "take responsibility for the environment at their rallies."

When Trump warned that there'd be "riots" at the GOP convention if he were denied the party's nomination, Ryan was sharply critical , saying that "nobody should say such things in my opinion because to even address or hint at violence is unacceptable."

But as recently as last week, Ryan was downplaying his rift with Trump. He'd encouraged Republicans to attend the party's convention in Cleveland, and said he'd had a "very pleasant conversation" with Trump.

"I feel like we will be able to unify Republicans and conservatives to offer the country this fall," Ryan told CNN , "a very clear and compelling choice so that the people of this nation get to decide where we go as a country."

A third-party candidate?

Conservative blogger and #NeverTrump movement leader Erick Erickson told CNN he and other Trump opponents are searching for a candidate who could mount a third-party bid against Trump and Clinton.

"Donald Trump cannot consolidate the Republican base and many Republicans cannot accept a Hillary Clinton donor as the Republican nominee," Erickson said. "If the delegates ratify this madness in Cleveland, many of us will look elsewhere for a credible candidate to oppose both Trump and Clinton."

Freshman Nebraska Sen. Ben Sasse has emerged as a central figure in the movement to oppose Trump. He argued for a third-party candidate in a Facebook post early Thursday morning.

"Why shouldn't America draft an honest leader who will focus on 70% solutions for the next four years?" Sasse wrote. "You know ... an adult?"