About six years ago, one of my producers at the Fox News Channel received a call out of the blue in which the caller asked if I'd be interested in coming on The Colbert Report. At the time, I was generally unfamiliar with the genre of late-night comedy because I am usually in dreamland when it airs.

Colbert was hilarious in his haughty imitation of the excesses of my Fox colleague Bill O'Reilly. I did not and do not know what his personal politics are, but his shtick was playing O'Reilly to a T. In fact, at one point during our back-and-forth, I inadvertently called him "Bill." He stopped the act, came out of character and kissed me. Later he claimed it was one of the most novel and show-stopping uses of a single word he had experienced.

A few months later, my producer received a call from a producer for Colbert's boss, Jon Stewart. That caller asked if I wanted to be on Jon's show to discuss my latest book. That was five books and five appearances ago.

The on-air experience with Jon Stewart is unlike anything I do at Fox. First, when you are Jon's guest, you are his only guest. Second, as the only guest, you are on the show for about 15 minutes. In TV time, and by Fox standards, 15 minutes as an on-air guest is an eternity. Third, Jon spends the first 15 minutes of the show bashing you or your ideas or your employer—and he did this to me all five times I was on with him. So, by the time I first walked onto his set, the 400 or so folks in his live studio audience were expecting me to have two horns and a tail.

I decided to push the envelope a bit. Before Jon could question me, I told him my purpose in coming on his show was to out him as a closet libertarian. "Go ahead, Your Honor. Give it a try. This is ridiculous." OK, here goes… "Is it any of the government's business whom you marry or what you put into your body or to whom you speak on the phone?" "Of course not." Good. "Should the U.S. be traveling the globe looking for monsters to slay?" "No; never." Excellent. "Can the feds constitutionally capture the content of your phone calls and emails without a search warrant from a judge based on evidence about you?" "No." Terrific.

I should have stopped there, but I was having too much fun with America's funniest comedian. "Are you a secret admirer of Rand Paul?" OK. I blew it. He exploded with his characteristic inanity, and the audience roared with Jon and me, thinking we had rehearsed this.

So began a happy friendship during the course of which we have discussed and analyzed on air nearly every aspect of the proper role of the government in our lives. We were and are admittedly an odd couple. He is the progressive Jewish funnyman with a tongue as sharp as a serpent's tooth, trying to sound smart. I am the libertarian traditionalist Catholic defender of the Constitution, trying to be funny.

Well, a strange thing happened. We agreed more than we disagreed, and I learned what most of his fans do not know: Jon does not need to try to sound smart; he may be the smartest personality on television. He is as well read in the works of those with whom he disagrees as he is in the works of those who reinforce his own views.

During last winter's version of the polar vortex, a bit like the weather this week in the northeast, I left a restaurant in lower Manhattan one night and was waiting for a car to pick me up when I saw a man wearing two overcoats and a hoodie walking a three-legged dog. I couldn't see the man's face, but I recognized the dog, and the dog recognized me. Then I heard the man, who was on his cellphone, spell my last name in a loud booming voice to whomever was on the other end of the call.

It was Jon. His dog jumped up to lick my face. Jon hugged me and said, "Judge, please come on the show next week. Let's talk about something that we will not disagree on." "What?" "Lincoln…" Oy. We disagreed profoundly. Jon had a black actress about a foot and half taller than I am playing Lincoln and three leftie-loony professors who were blind to historical truths trying to rough me up. It was hilarious.

I was saddened when Jon announced his retirement last week. Jon is a comic genius. His ratings are the envy of all in cable TV. He is the principal news source for millions of young people, who are the principal targets of TV advertisers—and God love him, he is his own boss. He has five dogs who run around that studio, and also a lot of love. Agree with him or not, he is intellectually honest, sharp as a tack, and so warm and engaging that you just don't want your time with him to end.

I hope mine doesn't.