Mike Downey is a former Los Angeles Times and Chicago Tribune columnist and a frequent contributor to CNN.

(CNN) Jerry Seinfeld joked on his TV show that loyalty to athletes is strictly a matter of laundry. You love a guy who wears your favorite team's uniform, but not if he puts on a different team's shirt. Or, as Seinfeld summed it up: "You're actually rooting for the clothes."

It seems like a safe assumption today that a true-blue fan of anyone in a New England Patriots uniform believes that the team's quarterback, Tom Brady, is being unjustly treated by the nasty National Football League.

In contrast, someone with no emotional commitment to that team might feel its "superhero" is a ballsy cheater who the "Justice League of America" -- the completely neutral NFL -- caught red-handed.

"Safe assumptions" are at the very heart of the NFL vs. Brady case, which has resulted (so far) in the quarterback of the Super Bowl's defending champions going directly from that game of thrones to being banned from the first four games of the upcoming 2015 season. The NFL upheld the penalty Tuesday upon further review as punishment for using unfairly deflated (easier to grip) footballs in last season's 45-7 AFC championship victory in January over the Indianapolis Colts.

With no irrefutable proof being found, the NFL simply assumes that the quarterback is guilty as charged.

Mike Downey

Brady is irate. So is Robert Kraft, who owns the team. And so are their Patriot exponents and obedient servants, who take Brady at his word that he wasn't responsible for those too-soft balls.

Brady: "I did nothing wrong."

Kraft: "I was wrong to put my faith in the league."

Patriot fans: "I (Heart) Tom!"

Nonbelievers -- principally Roger Goodell, the NFL commissioner -- do not feel the same way.

"Mr. Brady had failed to cooperate with the investigation," said Goodell.

His denials? "Not credible," said Goodell.

Above all, why did Brady let his cell phone -- which had sent out about 10,000 texts in the four months he was using it -- be destroyed, knowing full well the NFL's investigators were waiting to see it?

Brady's doubters: So he can hide his guilt.

Brady's boosters: So he can keep his privacy.

Brady himself: "There is no 'smoking gun,' and this controversy is manufactured to distract from the fact that they have zero evidence of wrongdoing."

The league's point of view is that Brady's phone destruction, as the commissioner's statement put it, supported the NFL's already-stated conclusion that he "sought to hide evidence of his own participation in the underlying scheme to alter the footballs."

So, here we go again then, sports fans -- weighing what we think against what we know.

The baseball player we assume used steroids versus the one who actually flunked a drug test. The former football player we assume committed murder as opposed to the one a jury didn't convict. The bike-racing hero we believed in or was suspicious of, versus the one who got exposed as a fraud and a liar and then confessed.

Welcome to another episode of "Catch Me If You Can."

Will the real Tom Brady be revealed? No, not bloody likely, because the evidence is circumstantial, so the pro-Toms can stand firm that he would never knowingly cheat, while the anti-Toms can continue to maintain that, "Oh, come on, the guy cheated and the Patriots cheated and the NFL knows it and you know it and I know it and he knows it and they know it."

Guilty as charged. (Just not proved).

The rub is that professional athletes abide by the arbitrary disciplinary decisions of their leagues. Practically speaking, the verdict the NFL handed down to Brady cannot be entirely reversed. A reduction in his four-game suspension is possible, perhaps, but no way could it be reduced from four games to zero.

Brady is accountable, period, in the NFL's eyes.

After the NFL issued its investigative report on Deflategate in May, even the team's owner, Kraft, said he wouldn't challenge whatever penalty the league imposed.

But it turned out Kraft, too, believed in assumptions -- his being that the NFL would think things over and give Brady a break.

Now he is livid. Kraft says it is "incomprehensible to me that the league continues to take steps to disparage one of its all-time great players," and that the league is tarring a 37-year-old player's spotless reputation with "implied nefarious behavior."

Brady's defenders can rant and rave, but the fact is, the NFL spent countless hours and dollars investigating "one of its all-time great players," showing him no favoritism, and had no explicable reason to want to find him guilty. What ulterior motive could the organization possibly have had in ushering one of its most popular personalities off the playing field? In banning him for a fourth of its championship team's 16 scheduled games for no justifiable reason? That would be borderline insane.

Brady does his cause no favors. The claim, "I did nothing wrong and no one in the Patriots organization did, either," is a broad, sweeping and preposterous statement.

Those improperly inflated footballs were not delivered by evil elves from Santa's toy shop or invisible wizards from the oblong-pigskin galaxy. Was he implying that a mysterious stranger with no connection to the Patriots (and in possession of a fake ID) sneaked into the stadium pushing a fresh shipment of balls in a wheelbarrow?

The NFL, according to Brady, judged that it was "more probable than not" that Brady was "at least generally aware" of something improper, and therefore deserved punishment.

So what texts could have been on that phone of his?

"NFL, WTF!"

"Hahaha, I fooled U."

"Pats R 2 Smart."

"I Can Pass, But U Can't Catch Me!"

OK, probably not.

It could be Brady destroyed his phone because his Samsung was old and Apple was having a big sale. Or, it could be Brady destroyed his phone because it contained photographs of his glamorous supermodel wife along with his hoodie-garbed coach, modeling each other's wardrobes. Or it could be Brady got a text from Hillary Rodham Clinton that read: "Trust me, Tom, get rid of it!"

Whatever was on there, it's in pixel heaven now. Brady did not bring his phone or its contents to his suspension appeal hearing in June, as some in the NFL obviously were expecting him to do. Instead, he committed Samsung suicide and had its dead cells sent to That Big Phone Boneyard in the Sky, where old texts go to die.

He is guilty now because the NFL says he is guilty -- period.

Friends are free to defend him. Foes are free to condemn him. And the one absolutely sure thing is this: If he wore a Colts or Jets or Cowboys or Broncos or 49ers jersey for a living, a whole lot of Patriots fans today would want his suspension to be even longer.