(CNN) National Security Adviser H.R. McMaster is increasingly isolated in President Donald Trump's administration, congressional and administration officials tell CNN, finding himself in conflict with senior White House officials, Cabinet members and the President himself.

A congressional source tells CNN that McMaster is at odds with President Trump on many key national security issues. McMaster has also found himself undercut by others in the President's orbit like chief strategist Steve Bannon and he has clashed with Defense Secretary James Mattis, according to congressional and administration officials.

McMaster's dust-ups with the President and his team have bubbled up over a long-delayed plan for Afghanistan, where the President has yet to sign off on a McMaster-backed strategy that would increase the number of US troops deployed to bolster America's longest war.

As the dispute has dragged on, administration officials have speculated that McMaster could be on his way out of the White House, potentially taking a new job within the administration or military outside of the National Security Council.

McMaster met with Trump in the Oval Office on Thursday as the NSC looks to finalize an Afghanistan strategy that the President will sign off on.

CNN has spoken to a number of government officials who say several scenarios are possible in the coming weeks, including the unlikely possibility McMaster could be transferred to command troops in Afghanistan or tapped as a new "special envoy" for South Asia, where he would oversee policy and strategy for Afghanistan, Pakistan and India. A senior administration official dismissed the notion as premature and not under serious consideration.

McMaster staying put as the national security adviser, the sources say, will depend largely on how comfortable Trump feels keeping him on in his current position amid all the disarray in the White House.

However, two intelligence officials caution that the warnings about McMaster's fate that have arisen in recent weeks may originate from other factions in the administration trying to undermine the national security adviser's authority.

"Bannon's been undercutting him and NSC, but that's nothing new," said a Republican congressional aide. "It's Game of Thrones over there."

The drama surrounding McMaster comes amid reports several others in Trump's Cabinet are feeling frustrated and could be on the way out.

Trump has taken the extraordinary step of attacking his own Attorney General, Jeff Sessions, on Twitter in recent days. The President said Tuesday he was "disappointed" in Sessions but would not say if he would fire him, adding only that "time will tell" Sessions' fate.

Two sources told CNN on Sunday that Secretary of State Rex Tillerson could be on the way out, too, amid his own issues with the White House. The State Department has pushed back, and Tillerson said Wednesday he'll stay "as long as the President lets me." He added that his relationship with Trump is "good."

McMaster has clashed with Bannon over a troop increase in Afghanistan for several months, as they have competed to win over Trump on the issue. The dispute has sparked leaks about McMaster losing his standing within in the White House, including a Politico report Monday that Trump's hesitation to add troops in Afghanistan was a no-confidence vote in his national security adviser. An NSC Principals Committee meeting on Afghanistan last week was heated and "words were exchanged," according to Politico.

McMaster has expressed concerns about Russian aggression, as has Mattis. But as a longtime proponent of military counter insurgency, McMaster may have run afoul of the White House interest in limiting large scale military involvements and instead focusing on the use of special operations forces.

The National Security Council declined to comment on the record for this article. Steve Bannon did not respond to a request for comment.

Closer to a decision?

The National Security Council's Principals Committee met again Wednesday to debate US policy in South Asia, including the future of the US effort in Afghanistan, administration officials said.

Pence chaired the meeting instead of McMaster, and he was there to ask questions and help guide the discussion, according to the officials.

One official pushed back on the notion McMaster is isolated, insisting that McMaster and other members of the principals committee have worked together to secure the President's support on major decisions like granting the Pentagon additional troop management authority and recertifying Iran's compliance with the nuclear agreement.

Another senior administration official told CNN the meeting primarily focused on addressing the concerns Trump has expressed in recent weeks about the US's long-term commitment to the war in Afghanistan and his questions about what success would look like there.

The officials established barometers to measure success as the US presses forward in the 16-year war, and claimed the top national security and foreign policy officials emerged from the meeting united on ways to address the President's questions and concerns on the US's ongoing commitment to Afghanistan.

The official said that while chief strategist Steve Bannon, who has feuded with McMaster over the strategy in the past, attended the meeting, he did not speak up.

Problems at the Pentagon

McMaster joined Trump's White House in February as an active-duty three star general replacing Michael Flynn, a retired three-star general. The move put McMaster in an awkward place with Flynn's allies in the White House, which was exacerbated when McMaster removed Flynn loyalists from the NSC -- and got Bannon removed from the principals committee.

Retired Col. Derek Harvey, a Flynn appointee to the NSC, was removed from his post on Thursday.

But internal White House politics aren't McMaster's only issues in the administration, as Mattis has also grown frustrated over the delay on an Afghanistan decision and the two have clashed over it, according to Republican lawmakers and administration officials.

"The biggest problem is he and McMaster have had these tensions," said a Republican senator. "That's the problem -- over roles, missions, budgets. Why do you think we haven't got a strategy yet?"

Defense officials also pointed out that as a three-star general, McMaster is essentially outranked by both Mattis, a retired four-star Marine general, and Gen. Joseph Dunford, chairman of the Joint Chiefs of Staff.

While McMaster has immediate access to the Oval Office, he is more of a coordinator than policymaker in the eyes of the Pentagon, defense officials say.

Mattis, they say, has no hesitation about going directly to Trump when he feels it's necessary.

Of course, the Afghanistan strategy delay also underscores Mattis' own difficulties with Trump. Mattis had promised Congress a plan by mid-July that has still yet to materialize. And Wednesday's decision to ban transgender troops caught nearly everyone in the Pentagon off guard -- including the Joint Chiefs -- and came while Mattis was on vacation.

One military official who has worked with McMaster says he has made some at the Pentagon uncomfortable for publicly taking what they see as political positions in defending the President, including when he publicly defended Trump after his Oval Office meeting with the Russian ambassador and foreign minister and disputed reports that the President had shared classified information.

Mattis dismissed reports that he and McMaster are at odds, or that McMaster could be on his way out at NSC.

"We love reading those kinds of stories going -- I'm amazed sometimes when I'm having a fight with and everything, somebody I haven't talked to in two weeks. I'm having a fight with them. It's good. You know, it keeps us smiling," Mattis told reporters last week.