Steve Bannon's position as President Trump's chief strategist may be in jeopardy, two sources close to the White House told CBS News' chief White House correspondent Major Garrett.

There is a growing feeling both inside and outside the White House that Bannon is increasingly falling out of favor with Mr. Trump, with the relationship between the two falling to all-time lows.

One source says that Bannon could be gone by the end of the week, Garrett reports.

Get Breaking News Delivered to Your Inbox

Over the weekend, White House officials were not quick to come to Bannon's defense on the subject of his loyalty to the president and the White House's agenda.

On Sunday, White House National Security Adviser Gen. H.R. McMaster would not say whether or not he would work with Bannon during an interview on NBC's "Meet the Press." Breitbart, the right-wing news service that Bannon ran before joining the Trump campaign, has run numerous articles critical of McMaster in recent weeks.

When asked what was going on between him and Bannon within the White House, McMaster responded that "there's just a lot of noise, from my perspective."

"And I think everybody sees the president has assembled a tremendous national security team," McMaster said. "And it's a great privilege for me to be able to support and enable people like Secretary of State Tillerson, Secretary of Defense Mattis, [CIA] Director Pompeo, Ambassador Haley at the United Nations. And despite this noise, we're getting quite a bit done in terms of developing and advancing strategies to prioritize the safety and security of the American people and promote American prosperity."

In an interview with ABC's "This Week" on Sunday, former communications director Anthony Scaramucci questioned Bannon's loyalty and did not dispute the idea the president may soon fire him.

"Well, I think the president knows what he's going to do with Steve Bannon," Scaramucci said. "Well, let's leave it up to the president. It's his decision. But I mean, at the end of the day, I think the president has a very good idea of who the leakers are inside the White House. The president has a very good idea of the people that are undermining his agenda that are serving their own interests."

Adding to the tension, reports circulated last week alleging that Mr. Trump suspects Bannon of leaking sensitive information to the press. Just last month, Mr. Trump emphasized his frustration with these breaches of information and called on his administration to be "much tougher on the leaks."

Bannon may also have some problems with newly appointed Chief of Staff General John Kelly.

Kelly, who replaced former Chief of Staff Reince Priebus, is eager to reshape the Trump administration, and the president has deferred to Kelly for key staffing decisions. One source said that Kelly wants "to put more adults in place inside the White House," while also clearing the air of rivalries stemming from when former White House Chief of Staff Reince Priebus and Bannon were jointly appointed.