Washington (CNN) House Speaker Paul Ryan has the "unified Republican government" he campaigned for in recent weeks -- a GOP House, Senate and White House for the first time since 2006.

But as the realization sinks in that Donald Trump is the president-elect, the focus turns to the next election --- Ryan's own bid in a week to gain another term as speaker. And should he win, the question of how much power the Wisconsin congressman will enjoy.

There is some lingering criticism about Ryan's handling of his relationship with Trump as the speaker dials up members of his conference to secure their votes in the internal GOP election that is scheduled for next Wednesday. The last time the speaker addressed House GOP members he told them on a conference call he could no longer defend the Republican nominee after a video tape released depicting Trump speaking in vulgar terms about women -- a message that didn't go over very well.

Ryan was quick after Trump's victory to set aside the public feud with President-elect Donald Trump and pledge to work together.

"We are eager to work hand-in-hand with the new administration to advance an agenda to improve the lives of the American people. This has been a great night for our party, and now we must turn our focus to bringing the country together," Ryan said in a statement after Trump secured his White House victory.

One source close to discussions among members of the conservative House Freedom Caucus told CNN that if Ryan doesn't make a "big statement tomorrow to show he strongly supports Trump and will work with him," his job could be at risk.

But other Republicans are ready to move on and chart a course with a Ryan-Trump alliance.

North Dakota GOP Rep. Kevin Cramer, a strong supporter of Trump's, told CNN recently it was a mistake when Ryan announced he was turning his focus just to congressional races last month after the release of the "Access Hollywood" tapes. Although Cramer warned the speaker about a public break with Trump on that conference call, he told CNN last week, "certainly in my mind it doesn't disqualify Paul Ryan to be speaker of the House." Cramer is standing behind the speaker and expects him to win re-election to his post.

As the election returns came in on Tuesday night showing Trump heading to a victory, one senior member of the Freedom Caucus told CNN that a Trump win could "certainly help the speaker."

Other Trump supporters outside Congress may have leftover anger with Ryan, who even before that October break with the nominee, had an awkward and mostly arm's length relationship with the Republican presidential candidate.

Ryan initially held off officially endorsing the businessman when he won the party's nomination in May, and spoke out against some of his controversial statements on the campaign trail.

Will anyone challenge him?

Even before the election results came in showing Trump wins in unlikely areas, there were signs that Ryan could face some resistance in keeping his gavel. Several leading members of the conservative Freedom Caucus declined in recent days to say whether they would back Ryan for another term.

But no House Republican has launched a challenge against Ryan, and House GOP leadership sources are confident that he can win a majority in the secret ballot contest next Wednesday among House Republicans.

But they acknowledge that it only takes a small number of members to deny the speaker the 218 votes he needs on the House floor in January to officially be sworn in. While much of the focus has been on members of the Freedom Caucus, there is a bloc outside of that group who backed Trump and may be irked that the speaker didn't campaign with the candidate who won state after state he wasn't expected to carry.

Ryan tried to mollify his critics in the final days of the campaign - doing a series of local interviews in key House districts and on conservative radio programs in Wisconsin emphasizing his efforts to elect "a unified Republican government."

He rarely mentioned Trump's name during these interviews and instead unloaded a steady stream of attacks on Hillary Clinton, who he warned was a liberal who would the country through another series of scandals and lurch the government sharply to the left.

The day before the election Ryan stressed in a radio interview with a conservative host that a GOP-led Congress can quickly enact things with Trump in the White House like repealing Obamacare.

"We are on the same page with Trump on this," Ryan said. "We are on the same page as our nominee on this so when he says he wants to a special session, which means he wants to do it early, we are ready to go. We can do that."

The speaker defended the times he broke with Trump, saying "if I see conservatism being disfigured I'm going to stand up for conservatism."

But also emphasized on Monday "what matters here is that we come together. There are different kinds of conservatives, there are different kinds of Republicans. What matters is we unify and all go vote, most importantly the last thing we want is a Hillary Clinton presidency."

Agenda going forward and working with Trump

On Wednesday afternoon, Ryan will again address rank and file GOP members on a post-election conference call hosted by Oregon GOP Rep. Greg Walden, who heads the House Republican's campaign committee.

Certainly Ryan will get credit for his tireless work to keep the House GOP majority. He raised more than $40 million, shattering fundraising records, during the last year as speaker and traveled across the country to stump for incumbents and challengers. Most political analysts predicted the large House majority would shrink, but the speaker's work helped keep those losses down to single digits.

The question is whether Ryan's efforts in the final days of the election to appear more supportive of Trump will help insulate him from Republicans who are upset about how the speaker treated the incoming president.

Trump spent little time during his campaign laying out policy details so he will gain an eager partner to tackle major priorities that have been out of reach under the last eight years of the Obama administration. The president-elect could lean on the speaker's expertise on these issues, but he would need to get past the bad blood he had with Ryan, who he just weeks ago called "weak and ineffective."

One key bridge to smooth over the relationship is RNC Chairman Reince Priebus, who is a longtime friend of Ryan's from Wisconsin politics.

Ryan's close relationship with Mike Pence, Trump's running mate could also help him, although Pence last week declined to say in an interview three times that Ryan should be re-elected speaker. After that awkward development Pence did travel to Wisconsin and affirm his support for the speaker, a friend and former colleague from the House.

Ryan, even before the contentious primary process was over worked to craft a detailed "Better Way" agenda, and secured the billionaire businessman's support for it. The speaker argued that the roadmap would give a Republican president plans to address tax reform, overhaul of health care programs, and launch regulatory reforms.