(CNN) As Washington awaits a potential reshaping of the political order on Capitol Hill, multiple sources inside and outside the Trump administration tell CNN that an all but certain shakeup is quietly underway at the White House, as President Donald Trump is poised to reshuffle his Cabinet for the second half of his first term.

Trump acknowledged as much on Monday. "Administrations makes changes, usually, after midterms. And probably we'll be right in that category as well," he told reporters at Joint Base Andrews on his way to a rally in Ohio. "I think it's very customary."

The President has made little secret of his furor at several members of his Cabinet, with Attorney General Jeff Sessions at the top of the list. His relationship with Defense Secretary James Mattis has steadily faded, with the two talking far less than they did during the first year of Trump's term. And his anger toward Homeland Security Secretary Kirstjen Nielsen has bubbled into public view on several occasions, with the President blaming her for failing to solve the intractable challenge of illegal immigration.

As the halftime mark of his first term approaches, the President is increasingly talking to allies and advisers about the potential for sweeping change in his Cabinet. While as many as half a dozen may end up leaving, three names in particular-- Sessions, Mattis, Nielsen -- are at the top of many internal conversations about the Cabinet secretaries most likely to leave, along with White House chief of staff John Kelly. Another is already on her way out. Last month, US Ambassador to the United Nations Nikki Haley announced her intent to leave at the end of the year.

Even so, two White House officials and one outside confidant to the President tell CNN that the changes may be less urgent than many people assume. The timetable may also be dictated by the outcome of the midterm elections. "No one leaves quickly around here," one official said, speaking on condition of anonymity to discuss deliberations inside the West Wing. "Even people the President wishes were gone."

Jeff Sessions

That especially applies to Sessions, whom Trump has spent months openly ridiculing because of his recusal from the Russia investigation, issuing blistering performance reviews in media interviews and tweets, and even going so far as proclaiming, " I don't have an attorney general ," during an interview with The Hill in September.

For his part, Sessions has generally declined to spar with the President, choosing instead to keep his head down and go about his job while continuing to support Trump's policies (both privately and publicly). He's also managed to adopt a stoic sense of humor about his situation. To mark the anniversary of his first year in office, staff aides gave Sessions a bulletproof vest as a gag gift. Many Republican lawmakers have said Sessions should stay put. And Trump's attacks haven't gone over well with voters in Alabama, where Sessions remains popular.

Still, Sessions constantly finds himself in the cross-hairs of Trump, most recently during an Axios interview in which the President said Sessions did not give him prior notice before the Justice Department urged a court to throw out provisions in Obamacare that protect pre-existing conditions. Not only is that at odds with Trump's recent rhetoric on the campaign trail promising to protect patients with pre-existing conditions, but Sessions also has said that Trump signed off on the move.

It's all led to widespread speculation that Sessions could be the first to go. His departure would trigger a complex set of options for the President. If Trump does not immediately name a temporary replacement, then Deputy Attorney General Rod Rosenstein, the current No. 2 at the Justice Department, would act in Sessions' place. Given that Rosenstein has already stepped into Sessions' shoes to oversee special counsel Robert Mueller's investigation, allowing Rosenstein to serve as the acting attorney general until Trump's permanent choice is confirmed could provide the least disruptive path forward.

Senate leadership has privately signaled that another attorney general nominee would not get confirmed during the lame duck session after the midterms, according to a source familiar with the matter, further curtailing Trump's ability to quickly swap out Sessions.

If Trump opts to tap someone else as acting attorney general in lieu of Rosenstein before Mueller's probe is finished, the political blowback could be strong, especially if the move is perceived as a way to undermine the special counsel's work.

In the wake of reports that Rosenstein may resign or be fired in late September, Justice officials began planning for Solicitor General Noel Francisco to step in as acting attorney general to oversee Mueller. They also made plans to name Matthew Whitaker, Sessions' current chief of staff, as the acting deputy attorney general. And while Whitaker has spoken with Trump directly about his future at the Justice Department, reportedly even contemplating taking his boss's job , the White House has been informed that he too would likely have to recuse from the Russia investigation in light of some of his past writings and TV appearances. Plans of his ascension appear to have stalled. Moreover, there seems to be little appetite for Francisco to assume the acting attorney general slot if Rosenstein remains in place for now.

One Trump ally told CNN that the White House is now looking for someone who is "not real partisan" when asked about candidates to permanently replace Sessions. The source acknowledged the swirl of rumors that it might be a sitting lawmaker -- Republican Sen. Lindsey Graham of South Carolina has been repeatedly asked about the job. He's also repeatedly said he's not interested.

Other names that have been floated as possible replacements for Sessions include Rep. John Ratcliffe of Texas, a former federal prosecutor, as well John Michael Luttig, a former US Court of Appeals judge and Justice Department official, and current general counsel at Boeing. One former White House official said Luttig is a strong possibility. A second source close to Luttig says that he was considered for the role of FBI director after James Comey was fired.

A Hill source also mentioned that federal Judge Edith Jones, who sits on the 5th US Circuit Court of Appeals, might also be in play.

Janice Rogers Brown has also emerged as a possible candidate. Brown, a former California Supreme Court associate justice and former US circuit judge, retired from the bench in August 2017 and moved back to California. A source close to her indicated that she would not be interested in the AG job and doesn't want to return to Washington.

John Kelly and Jim Mattis

One of the biggest dynamics to watch -- and one of the biggest mysteries in Washington -- is between Kelly and Mattis, both retired Marine Corps generals, and whether they have an unspoken agreement to stay as long as possible. Two questions remain unanswered for several people close to the White House: Will they stay on until fired? Or will they resign and let the President shape his team for the next phase of his first term?

In July, Kelly made a showy declaration that Trump had asked him to remain as chief of staff until 2020, and that he'd agreed. But even that pronouncement did little to quiet speculation -- both inside and outside the White House -- that he could depart at any time.

Mattis' future was brought into question most recently after Trump, in a September interview with "60 Minutes," labeled Mattis as "sort of a Democrat" while saying the retired general "may leave" and that "at some point, everybody leaves."

CNN in recent days has spoken to several current and former officials who have worked directly with and for Mattis. They all say he would not resign because of the sense of duty he feels to the troops. One current senior administration official who works directly with Mattis says the defense secretary still sees himself as a general and that since the troops can't just up and leave, he won't either.

Mattis himself pushed back on rumors of his departure, telling reporters while en route to Vietnam recently that Trump had called him following his interview with "60 Minutes" and told him, "I'm 100% with you."

"I'm on his team," Mattis told reporters, adding, "We have never talked about me leaving."

On Monday, Trump directly addressed speculation about Mattis leaving, and denied he had any plans to change his secretary of defense. "Why would I do that? Is that the new rumor?" he asked when questioned about changes at the Pentagon. "No, I don't at all. I was surprised by that question."

The one development that could change that line of thinking by Mattis would be if there were so much interference by the White House that he felt he could not run the Defense Department. These officials who have worked with Mattis say a key hurdle for him will be to convince the President that his ideas are not being slow-rolled at the Pentagon. As one official said, "That is why you see the deployment of forces" to the border that the President wants. "There were the '60 Minutes' comments and now you see Mattis carry out the President's orders." On Oct. 31, asked if the President's plan to send thousands of troops to the border amounted to a political stunt, Mattis replied, "We don't do stunts."

Some current and former officials have said that rumors of Mattis falling out of favor with the President are being pushed by allies close to Trump's new national security adviser, John Bolton, specifically his deputy Mira Ricardel, who clashed with Mattis while serving in the Trump transition and worked to block some of his choices for key Pentagon posts in the early days of the administration.

Bolton has also clashed recently with Kelly, including getting into a West Wing shouting match with him over immigration policy and a recent surge of illegal border crossings.

When news of the loud and expletive-laden argument with Bolton emerged, some White House staffers wondered whether Kelly would step down -- a reflection of the continued questions about his status.

That Bolton felt empowered to raise his voice to Kelly struck some staffers as a sign of Kelly's diminished role. When he took the chief of staff job in July 2017, Kelly insisted that all aides -- even Trump's daughter Ivanka Trump and son-in-law, Jared Kushner -- report through him. But new additions to the staff, including Bolton, top economic adviser Larry Kudlow and communications chief Bill Shine, report directly to Trump.

Kirstjen Nielsen

The performance of Homeland Security Secretary Nielsen was at the heart of the fight between Bolton and Kelly, two sources familiar with the argument told CNN at the time. Nielsen is widely viewed as an acolyte of Kelly, whom she served under at the White House and later replaced as secretary of homeland security. Kelly has staunchly defended Nielsen against criticism of her performance on immigration-related matters. He's also been forced to defend her to the President, who has expressed suspicion over the jobs she held in former President George W. Bush's administration.

Trump has angrily aired his frustrations with Nielsen's handling of border security during contentious meetings at the White House, claiming she isn't up to the task of fulfilling the campaign promises he made to curb illegal immigration. That, in turn, has led to an internal dynamic where some officials have griped to the President about Nielsen's performance in order to gain favor with him.

The President has not sought to quell that dynamic, believing that pitting camps of aides against each other is a way to produce better results. But the constant arrows have led Nielsen to tell some associates that she is unhappy in her post, leading some to believe she could be out of office as soon as the end of the year, according to one current and one former Department of Homeland Security official. The former official expects the decision will be mutual and that she will go to the private sector.

Asked if Nielsen was planning to stay in her role, DHS press secretary Tyler Houton said, "The secretary is honored to lead the men and women of DHS and is committed to implementing the President's security-focused agenda to protect Americans from all threats and will continue to do so."

One White House official told CNN over the weekend that Kelly's and Nielsen's fates seem tied. If Kelly leaves, Nielsen will likely leave too, according to this person.

Ryan Zinke

Another Cabinet member who may be on thin ice is Interior Secretary Ryan Zinke, not because he's fallen out of favor with the President. In fact, the two have a strong affection for each other. Rather, Zinke is now facing a probe by the Department of Justice, according to two people familiar with the investigation. Its scope isn't known, but it comes as the inspector general's office has looked into a series of issues involving Zinke, including questions about conversations he had with Halliburton Chairman David Lesar, who is backing a development project in Zinke's hometown of Whitefish, Montana.

Aides on Capitol Hill have speculated that Zinke could step down after the midterms to run for governor of Montana. Though a political operative in the state told CNN there is already a growing list of Republican gubernatorial candidates lining up to run and there may be less room for Zinke to jump in. He could also position himself for another job in the Trump administration -- one where he could use his national security background. But Democrats warn that if they win a majority in the House, Zinke would be high on the list of Cabinet secretaries who would be brought before Congress to answer questions.

"I would find it hard to believe that he would be appointed to something that requires Senate confirmation, given his baggage," one Democratic Senate aide told CNN.

In the end, Zinke may be safe given his relationship with the President. "Zinke is like Pruitt: The President likes him. He will ride this out," a White House official said, comparing Zinke to Scott Pruitt, the former Environmental Protection Agency administrator who hung on for months despite an ethics scandal. On Monday, Trump said Zinke is doing "a very good job" but that he was still looking into the matter of Zinke's investigation.

An Interior Department spokesman pointed to Zinke's record as evidence that he intends to stay. "Despite what anonymous sources who have zero insight into the matter say, Secretary Zinke will continue to serve as President Trump's Secretary of the Interior," the spokesman said in a statement to CNN. "Sec. Zinke has been incredibly effective at executing the President's agenda such as increasing federal energy production and revenues (we just announced an increase of about $2 billion in FY18), cutting regulatory costs by more than $5 billion since taking office, and advancing legislation to improve infrastructure on public lands. "

The Silent Majority

People familiar with White House dynamics say serving in Trump's Cabinet can often feel like walking through a shooting range. Keeping a head down and avoiding the President's attention -- even when it's positive -- is often more beneficial to self-preservation than landing on his radar at all.

That's largely worked for less-noticed officials like Energy Secretary Rick Perry, Agriculture Secretary Sonny Perdue, Small Business Administration head Linda McMahon and Transportation Secretary Elaine Chao, who have all conducted their jobs without much fanfare and equally little attention from Trump.

Compare that to officials who found themselves embroiled in ethics scandals or overshadowing the President or both. The drawn-out firing of Pruitt was stalled for months by Republicans enamored with his anti-regulatory push, but he'd long lost the President's confidence amid negative headlines. Former Health and Human Services Secretary Tom Price was swiftly ousted when his private plane use began dominating headlines.

Despite a flap over the purchase of some high-priced office furniture earlier this year, Housing and Urban Development Secretary Ben Carson has managed to stay under the radar and in the good graces of the President. It seems likely that Carson will stick around. Syndicated columnist Armstrong Williams, a close friend and confidant of Carson's, tells CNN that Carson plans to say on through Trump's first term "because the President asked him to and he believes that's how much time it will take for his work at HUD to take hold and make a difference."

Education Secretary Betsy DeVos has also kept a low profile, despite making headlines for some controversial comments earlier in the term. One Republican Senate aide on Capitol Hill told CNN that DeVos is hoping to stay in her post and that she recently promoted Nate Bailey as her new chief of staff. Bailey, a former director for public affairs at the Chamber of Commerce, is seen as someone who could more effectively prepare DeVos for possible hearings if Democrats take the House.

Another Cabinet member who appears not to be going anywhere is former CIA Director and current Secretary of State Mike Pompeo. One source points out that Pompeo has emerged as Trump's clear go-to, right-hand man, putting him atop the pecking order inside the Cabinet. Pompeo could play a significant role in the coming months in settling ongoing disagreements between Bolton and other Cabinet members, including Kelly and Mattis.