Christopher Ruddy, the founder of Newsmax Media and close friend of President Trump, told PBS News Hour Monday that he believes the president is “considering perhaps terminating” special counsel Robert Mueller, the man charged with investigating Russian interference in the U.S. election and possible collusion with Trump’s campaign.

The comments come amid increasing frustration at the White House and among Trump supporters that the investigation will overshadow the president's agenda for months to come.

When reached by Fox News after the remarks, Ruddy said, “while I am not claiming the president said it to me, I am confident of my sourcing. He is definitely considering it as an option.”

A source close to the Trump legal team, though, later told Fox News that Ruddy’s statement just came from watching another Trump lawyer, Jay Sekulow, discuss the Mueller probe on ABC’s “This Week” – and Sekulow did not directly suggest the president was thinking about firing Mueller.

Sean Spicer, the White House press secretary, said Ruddy “never spoke to the president regarding this issue. With respect to this subject, only the president or his attorneys are authorized to comment.”

The New York Times reported that to fire Mueller, Trump would have to order Deputy Attorney General Rod J. Rosenstein to turn back regulations that protect the special counsel from being fired for no good reason. If Rosenstein refused, Trump could fire Rosenstein.

As Mueller builds his legal team, Trump's allies have begun raising questions about the former FBI director's impartiality, suggesting he cannot be trusted to lead the probe.

Former House Speaker Newt Gingrich, an informal Trump adviser, tweeted Monday, "Republicans are delusional if they think the special counsel is going to be fair. Look who he is hiring."

Just weeks ago, Gingrich had heaped praise on Mueller, hailing him as a "superb choice" for special counsel whose reputation was "impeccable for honesty and integrity."

But after the testimony of former FBI Director James Comey last week, Gingrich said he'd changed his mind.

"Time to rethink," he tweeted Monday, citing Mueller's hiring decisions and Comey's admission that he'd instructed a friend to share with reporters notes he'd taken of his private conversations with Trump in order to force the appointment of special counsel.

Conservative commentator Ann Coulter offered a similar message, tweeting, "Now that we know TRUMP IS NOT UNDER INVESTIGATION, Sessions should take it back & fire Mueller."

The talk about dismissing Mueller appeared to be coming from Trump allies — including some close to White House strategist Steve Bannon — who are increasingly frustrated with the prospect of a long and winding probe.

They say Trump did not collude with Russia and see the investigation as a politically motivated sham that handicaps Trump's ability to execute his agenda, according to one person who advises the White House on how to handle the probe. The person demanded anonymity to discuss strategy on the sensitive matter.

Ruddy appeared to be basing his remarks, at least in part, on comments from Jay Sekulow, a member of Trump's legal team, who told ABC in an interview Sunday that he was "not going to speculate" on whether Trump might at some point order Rosenstein to fire Mueller.

"Look, the president of the United States, as we all know, is a unitary executive. But the president is going to seek the advice of his counsel and inside the government as well as outside. And I'm not going to speculate on what he will or will not do," Sekulow said. Still, he added, "I can't imagine that that issue is going to arise."

Fox News' Serafin Gomez, Ed Henry and John Roberts and The Associated Press contributed to this report