At a recent town hall meeting at a National Security Agency facility in Texas, employees at the U.S. government’s biggest and most secretive intelligence bureau told their boss, Adm. Michael Rogers, that they were concerned about how the agency’s reorganization was upending their work lives.

“His response was that employees should ‘stop bitching’ and do their jobs,” a recently-retired NSA analyst told The Daily Beast. That remark produced a “small outcry” that was discussed on an internal NSA blogging site where employees are encouraged to share their views. And it became symbolic to some of the tensions that have grown in the two-and-a-half years Rogers has been in charge of NSA.

Major policy disagreements between Rogers and top Obama administration officials—including ones about how hard to go after ISIS—have exacerbated long-standing grievances among NSA rank-and-file employees about Rogers’s brusque and aloof management style. Now, those complaints are erupting on the surface in a manner that has surprised current and former officials.

Most recently, The Washington Post reported an effort by the Secretary of Defense and the nation’s top intelligence official to remove Rogers from his post, as well as from the command of U.S. Cyber Command, which conducts cyber warfare operations.

It’s highly unusual for these sorts of internal struggles to emerge into the open press. But it’s even more unusual for a serving officer to do what Rogers recently did—meet with the president-elect about a job without notifying the current secretary of defense or the White House. Rogers huddled with Donald Trump last Thursday and could be in line to be named director of national intelligence in the next administration, if the reports out of Trump Tower are to be believed.

No one has yet described a particular offense by Rogers worthy of firing him. But in interviews with current and former officials, it became clearer that Rogers has lost support both among senior Obama officials as well as the employees he leads.

Two former intelligence officials, speaking on condition of anonymity, said that Rogers has never been embraced by NSA employees and has on various occasions alienated them.

“I think his leadership style was very much born of the military world. He tells you what to do and you do it. You don’t question,” the former analyst said.

“The problem is he’s leading a bunch of civilians at NSA who are free thinkers, and require critical thinking to do their job.”

The former analyst said that complaints about office life and work-life balance have been a frequent subject of discussion around the water cooler and in meetings with top agency leaders.

The grievances ranged from menial matters like insufficient parking to restrictions on whether employees could bring spouses to so-called morale building activities, like outings to baseball games.

But a recurring subject of rancorous debate was a reorganization of the agency that Rogers led. On a high-level, the reorganization has set up whole new structures, including an operations directorate, that Rogers believes will prepare the agency for the new threats it faces, like terrorists who encrypt their communications or foreign government hackers trying to disrupt U.S. elections.

But as a result of the reorganization, some people’s day-to-day lives have been turned upside down. Positions were eliminated while others found themselves suddenly reporting to new managers, the two former officials said.

The tumult of the reorganization—which was perhaps to be expected in an agency that employees tens of thousands of people—exacerbated already rock-bottom morale since the 2013 leaks of classified information by contractor Edward Snowden. The leaks were a major source of embarrassment for the spy agency and led to internal restrictions on access to information that some have said made them feel more like suspected potential leakers than trusted employees.

Current and former officials said that Rogers has also irked his more senior colleagues by missing meetings and frequently being away from the office without explanation. That stands in marked contrast to the management style of Rogers predecessor, Gen. Keith Alexander, who was beloved by the workforce and made a point of visiting with line-level employees and inquiring about their work.

Alexander endeavored to learn people’s names, remember details about their family, and try to relate to them on a personal level, former agency employees have said.

But the most significant clash Rogers has had may be with his boss, Secretary of Defense Ash Carter.

Within weeks of taking office in February 2015, Carter was focused on cybersecurity and cyberwarfare as one of his main policy interests. His first visit to a domestic military installation was Ft. Meade, Maryland, where the NSA and Cyber Command are headquartered.

That interest only intensified in the spring of 2015, when a U.S. raid led to the death of ISIS leader Abu Sayyaf and confiscation of ISIS computers that provided a treasure trove of information. Suddenly, intelligence analysts were sifting through ISIS documents and finding records of members’ communications. It was the makings of a roadmap for penetrating the group’s command structure.

Shortly after that, there were renewed calls from the Pentagon to ramp up cyber operations against ISIS. One U.S. official told The Daily Beast that Carter wanted Rogers to more aggressively hack ISIS so that it could not spread its propaganda worldwide or recruit new members. U.S. hackers did launch a campaign against the terrorist group aimed at disrupting its communications networks and trying to mislead fighters with false information. But the process stalled, and in Carter’s estimation, Rogers was not doing enough, the official said.

It’s unclear whether the men disagreed about a specific program or an overall approach. But Carter also had designs on splitting the NSA and Cyber Command, both of which Rogers’s currently directs. The internal thinking at the agency has been that the NSA would in the future be led by a civilian, which would put Rogers was out of the running for the job.

Rogers hasn’t been opposed to splitting the roles, but he has favored a slower approach, whereas Carter, joined by Director of National Intelligence James Clapper, has pushed to implement the changes more quickly.

With Rogers on the wrong side of a policy as his boss, and his support lacking among rank-and-file employees, he was in a politically perilous situation. One person familiar with internal discussions told The Daily Beast that last Wednesday, senior NSA officials “got wind” that the Post was preparing a story that would report Clapper and Carter’s recommendation to the White House that Rogers be removed from his position. The next day, Rogers met in New York with Trump.

Rep. Devin Nunes, the chairman of the House Intelligence Committee, told The Daily Beast that it was no coincidence that as soon as it became clear Rogers might have a role in the Trump administration, his critics have come out against him.

“This is even a new low even for these people,” Nunes said.

Rogers was asked about the controversy swirling around him during a panel discussion at the Halifax International Security Forum last week. He said only that he was “accountable for his actions.” When asked about his meeting with Trump, Rogers answered: “Next question.”