Washington(CNN) Top White House aides are worried FBI Director Christopher Wray could quit if the highly controversial Republican memo alleging the FBI abused its surveillance tools is released, multiple sources with knowledge of the situation tell CNN.

Wray has made clear he is frustrated that President Donald Trump picked him to lead the FBI after he fired FBI Director James Comey in May, yet his advice on the Nunes memo is being disregarded and cast as part of the purported partisan leadership of the FBI, according to a senior law enforcement official.

Wray's stance is "raising hell," one source familiar with the matter said.

Wray has not directly threatened to resign after clashing with Trump over the possible release of the memo, the source added, because that is not his style of dealing with conflict.

The potential release of the memo penned by House Intelligence Chairman Rep. Devin Nunes has set up a standoff with Trump against both the FBI and Department of Justice. Although the President has signaled that he is inclined to release the memo, as part of an effort to undercut the special counsel Robert Mueller's investigation, senior officials inside the White House are trying to come up with a solution that satisfies both the President and law enforcement officials like Wray and Deputy Attorney General Rod Rosenstein.

Multiple White House officials declined to comment. The FBI declined to comment.

One person briefed on the matter said the White House believes that redactions could cure the problem, given the White House plan is to approve the release of the memo and leave it up to Nunes' committee to officially release the document. But it is not clear redactions will be enough to satisfy concerns at the FBI, according to other officials, because their concerns are that substantial omissions make the document inaccurate, which is something redactions won't address.

White House aides expect a decision from Trump on whether to release the controversial memo on Thursday, a White House official told CNN, but the President has already made clear he is inclined to approve the document's release. The President has been subject to public and private pressure campaigns over releasing the memo and told a lawmaker who urged him to release the document on Tuesday not to worry.

Another person familiar with discussions about the memo said Wray didn't threaten to quit when he met with Kelly earlier this week and in numerous conversations since, but White House chief of staff John Kelly believes that is a real possibility and has been working on a way to avoid another departure from an already turbulent Trump administration.

Inside the FBI, which Trump has attacked as part of his effort to undercut Mueller's investigation throughout his first year in office, the feeling is that the Nunes fight could provide a small victory -- at least for now.

Agents inside the bureau are happy that Wray is standing up to the President, according to several law enforcement sources. The bureau released a rare statement on Wednesday that said the bureau had "grave concerns" with the memo.

"I think Chris Wray's motto is speak softly but carry a big stick," said one person inside the FBI. "One-point Chris Wray, zero points Devin Nunes."

There is a recognition, however, that Wray leaving could set off a chain reaction of events inside the law enforcement agency. Top officials inside the bureau have been trying to identify who might be considered a "Trump guy" in the order of succession in the bureau's organizational chart, another law enforcement official said.

Wray's current deputy -- David Bowdich -- was picked to be associate deputy director by Comey, who Trump fired in May. But Bowdich has a "straight shooter" reputation inside the bureau, and as a former Los Angeles executive at the FBI he differs from some of the recent leadership ranks drawn from Washington.

Trump's understanding of possible repercussions of releasing the memo is less clear.

The President continues to seethe over the Justice Department's handling of nearly everything Russia-related, sources said, but much of that anger has been directed toward Rosenstein, who oversees Mueller and the Russia investigation.