President Donald Trump's criticism of the FBI is getting some big time pushback.

But the people pushing back may have done more to hurt the FBI than anything coming out of the White House.

Just to recap, we learned this weekend that special counsel Robert Mueller removed a top FBI agent from his team after the Justice Department began examining whether the agent had sent anti-Trump text messages. The story became even more interesting when it turned out that same agent, Peter Strzok, helped lead the investigation into Hillary Clinton's use of classified information on her private email account.

That triggered the Trump Twitter machine with a number of inevitable comments:

It was the "FBI in tatters" tweet that encouraged the most compelling Trump opponents to respond. Former Attorney General Eric Holder, former acting Attorney General Sally Yates, and — most importantly — former FBI Director James Comey, all weighed in:

But each of these very similarly structured rebuttals actually made matters worse.

First, only the most obtuse observer would think President Trump's comments about the FBI were meant to include ordinary agents in the field. Comey et al seem to be intent on making sure the fight included as many people, innocent people, as possible.

Secondly, their comments completely glossed over the fact that the news of the Strzok removal was what President Trump was discussing. To be clear, this story constitutes a major image crisis for the FBI. Trump or no Trump, we now know there was enough evidence of blatant political bias to remove one of the most prominent agents working on two of the most important investigations of our time.

That's a big deal because Strzok had enormous reach for just one agent. Just since President Trump's tweet, published reports now say that Strzok is also the person who changed the key phrase in Comey's description of how former Secretary of State Clinton handled classified information from "grossly negligent" to "extremely careless." That change had significant legal ramifications that likely reduced the ability to prosecute Clinton.

After all that, Strzok was then put on the Trump investigation as a leader on Mueller's team. Didn't anybody worry about a possible conflict of interest by then? Wasn't there another good FBI agent available to share all these sensitive jobs? These are the questions damaging the entire FBI's credibility.

No, it's not fair to paint all the FBI's employees with that credibility cloud. But is that what President Trump was deliberately doing with his tweets? Probably not. Of course, that's where the president himself deserves real criticism. It's probably unrealistic to hope President Trump will somehow only start tweeting compliments and use softer language. But it's still possible to demand that he use his enormous power of public attention to be more exact and focus his fire on the right people. If President Trump wants to "drain the swamp," he needs to hone his attentions more precisely.

This tweet isn't the only sign of the administration's often too general and delayed efforts to make promised changes to the bureaucracy. The president's delayed removal of former IRS Commissioner John Koskinen was perhaps the most egregious example of that.

The result is that by being so imprecise, President Trump allowed Comey, Holder and Yates to deflect and even possibly obfuscate in their response. He could have avoided that by just focsuing on Strzok and Comey by name.

But the eerily similar responses from Comey and the others still feed the belief that top political appointees use the bureaucracy to carry out vendettas. This also increases the suspicion that they have more than a little to hide.

That's the kind of suspicion that sours people on all government. That's the kind of suspicion that hurts innocent federal workers. Oh, and most importantly for the president's Twitter opponents, that's the kind of suspicion that got Donald Trump elected in the first place.

Commentary by Jake Novak, CNBC.com senior columnist. Follow him on Twitter @jakejakeny.

For more insight from CNBC contributors, follow @CNBCopinion on Twitter.