Washington (CNN) There was a moment during Monday's coronavirus press briefing when President Donald Trump was asked about a recent government watchdog report about testing shortages in hundreds of hospitals across the US. As the reporter asked her question, a certain phrase triggered a reaction from Trump.

"Did I hear the word inspector general, really?" Trump said in a tone of annoyance. "It's wrong. It's wrong," he added, baselessly undermining the efforts of his own government to document how hospitals are dealing with the coronavirus outbreak.

Over the past few days there has been a significant escalation in the President's already unprecedented war against the oversight of his own administration -- and his sights are clearly set on inspectors general.

It started Friday, when Trump fired the intelligence community IG , Michael Atkinson, whose decision to bring a whistleblower complaint to Congress kicked off the impeachment inquiry last year. On Monday, Trump went after Christi Grimm, the author of that hospital testing report and the top official at the IG's office at the Department of Health and Human Services. He also falsely described her as an Obama appointee.

On Tuesday, in a series of bureaucratic moves, Trump removed Glenn Fine as acting IG at the Department of Defense. Fine had recently been named by a panel of federal IGs as chairman of a congressionally-mandated commission to oversee the $2 trillion coronavirus relief package.

Several sources tell CNN that Trump has long sought to remove Fine as the Pentagon's acting IG, viewing him as a holdover from the Obama administration. That effort escalated after Fine was selected by his peers to chair the pandemic response oversight panel.

Meanwhile, the new special IG in charge of policing a large chunk of the relief spending will be someone who works directly for the President, White House aide Brian Miller, who Trump nominated on Friday, around the same time he fired Atkinson.

In fact, much of the oversight apparatus in the relief package was a concession by Trump, who resisted Democratic efforts to create the commission and then waved away any true authority it holds with a signing statement attached to the bill.

"My Administration will treat this provision as hortatory but not mandatory," the President wrote.

House Speaker Nancy Pelosi told CNN's Anderson Cooper on Tuesday that the removal of Fine as the Pentagon's acting IG was a "problem" and a sign the President was seeking to put "some of his loyalists" in charge of overseeing how the relief money is spent.

The sequence of events reflects Trump's dim view of government oversight, particularly when it presents inconvenient facts. For Trump, who expects happy talk from all corners of his administration, the work of IG offices are a galling feature of what he often derides as the "deep state" -- a politically motivated permanent bureaucracy designed to thwart him.

"The system can't work this way," said former Treasury Department IG Eric Thorson, who was appointed by President George W. Bush in 2007 and retired last year. "Their work has to be respected. Hopefully, this won't change the way they do their work. They might be more fearful, but they'll still do the work. They're gutsy people, but it is hard. The system was not designed for this."

Suspicion of Obama loyalists

Trump has been skeptical of government watchdogs from even before he took office. Sources tell CNN that after winning election in 2016, Trump instructed officials to identify those IGs who were believed to be Obama-loyalists so that they could be removed and replaced with more Trump-friendly options.

In January 2017, White House staff phoned the Treasury and Labor department IGs to inform them they could soon receive pink slips, according to Daniel Meyer, who led the Whistleblowing and Source Protection Program at the Office of the Intelligence Community Inspector General until he was fired in March 2018 by then-Director of National Intelligence Dan Coats.

While those broader firings never occurred, Trump has maintained his suspicion of IGs throughout his time in office, multiple sources said. Those suspicions were amplified last year when Atkinson sent the whistleblower complaint to Congress. Since then, Trump has become increasingly fixated on removing IGs he views as holdovers from the Obama administration, according to multiple sources.

"It is not rocket science what the President is doing here. He has said publicly that he is the oversight authority and the IGs work for him," Meyer told CNN.

During his time in office, Trump has tweeted more than 25 times about various inspectors general and their reports, almost always using them to deride his opponents or to accuse IGs of bias, without ever providing credible proof.

He called Justice Department IG Michael Horowitz an "Obama guy" when it looked like he was slow-walking a review of surveillance abuses during the Russia investigation. But when Horowitz's report came out, and exposed serious wrongdoing , Trump hailed the findings and still touts them to this day.

During the impeachment saga, Trump accused Atkinson of perpetuating a "hoax" and colluding with Democrats by sharing the Ukraine whistleblower complaint with Congress, which he was required to do by law. Atkinson now says he believes his firing last week was retaliation for the Ukraine affair.

'Whistleblowing vacuum'

Even though they aren't well known, IGs have long served a crucial role in ferreting out waste, fraud and abuse inside the government. That role becomes even more crucial as the federal government begins doling out the record amount of coronavirus relief funds.

"Few Americans know what an Inspector General is, but we all suffer when they are punished for their independence," said Irvin McCullough, national security analyst for the Government Accountability Project.

"These watchdogs are entrusted to enforce whistleblowers' protections. If these watchdogs are attacked, defamed, and removed for their integrity, then the federal workforce will no longer trust them with their whistleblowing complaints. We will live in a whistleblowing vacuum," he added.

Even members of Trump's own party are urging him to cool down. Republican Sen. Chuck Grassley of Iowa, a staunch advocate for inspectors general and congressional oversight, responded to Trump's moves with a statement, reminding Trump that the law requires him to notify Congress ahead of any termination of a Senate-confirmed inspector general and to provide a written explanation.

"Inspectors General provide a critical check on an otherwise unaccountable bureaucracy," Grassley wrote. "In other words, they help drain the swamp."

Checks and Balances, the group of conservative and libertarian lawyers co-founded by George Conway, released a statement Monday charging that Michael Atkinson's firing underscored Trump's "contempt for the rule of law."

"While a president has the authority to nominate and select officials in his administration, he does not have the authority to make a mockery of the Appointments Clause of the Constitution, statutory mandates and Senate confirmation," the lawyers said.

Pandemic relief IGs most at risk

Many in the IG and whistleblower community believe those career officials on the Pandemic Response Accountability Committee are most at risk, including DOJ IG Michael Horowitz, who also heads the panel of IGs with the authority to pick the chair of that committee.

"People on the committee are likely at the top of list," according to a former attorney for the Ukraine whistleblower, Andrew Bakaj, who noted that he is particularly concerned about Horowitz's future. "Horowitz has proven to be independent and neutral, as he should be. Anyone who displays any sort of independence and neutrality is seen as a threat by this White House."

While Horowitz remains in his role, current and former officials caution that, as with any decision Trump makes, that could change with a single tweet. The fact that Horowitz has published sharply critical investigations into the FBI's handling of politically sensitive probes of Hillary Clinton and the President's campaign may help protect him, congressional sources and Justice Department officials tell CNN.

Horowitz has won high marks from key lawmakers including Grassley and Republican Sen. Lindsey Graham of South Carolina, and he also has a good relationship with Attorney General William Barr, officials note.

Horowitz is currently overseeing a deeper look at what he has said were serious concerns about the FBI's management of national security surveillance warrants under the Foreign Intelligence Surveillance Act. Trump has recently praised that work for highlighting the FBI's mistakes, an issue about which Trump has long complained.

Still, several IG postings have been occupied by acting officials after Trump nominees have been forced to withdraw from consideration for various reasons, and former inspectors general say that Trump's actions may have already muddied the waters in terms of the perceived independence of current watchdogs. Of course, Trump's battle with the IGs may end up costing more than just their independence.

"The American people are better off when the inspectors general are given free rein to audit and investigate," said Clark Ervin, a Bush appointee who issued hard-hitting reports as the first IG at the Department of Homeland Security. "The proof is in the pudding. Billions of taxpayer dollars have been recovered as a result of their efforts, which is far more than their collective budgets."