Jon Stewart has been the host of "The Daily Show" for 13 years, winning 18 Primetime Emmy Awards. But it almost never happened.

In an appearance with Stephen Colbert at a fundraiser in Montclair, N.J., on Friday for the Montclair Film Festival, Stewart recalled his initial conversations with Comedy Central executives before taking the reins from Craig Kilborn, the show's original host who left in 1999 to take over the "Late Late Show" on CBS.

"What I did not realize is, a lot of the people who worked there were assholes," Stewart said, according to Third Beat, an online magazine about comedy.

"I had, before taking [the job], some conversations with the powers that be there about the direction I thought we could move the show," Stewart continued. "I wanted it to be satirical in the classic sense of the word, not the Spy magazine sense of the word where you just add adjectives like 'pepperpot.'"

Stewart, who had been host of his own short-lived talk show on MTV in the mid-1990s, said his ideas were met with immediate resistance.

"I walk in the door, into a room with the writers and producers, and the first thing they say is, 'This isn't some MTV bullshit,'" he said. "And then I was told not to change the jokes or improvise."

Stewart said he called his agent to try to get out of the deal. "These people are insane," Stewart told him.

"How close were you to saying that was it, you weren't going to do it anymore?" Colbert asked.

"I had to be talked down from a moderately high cliff," Stewart said.

The comedian, who turned 50 last month, said it took about two-and-a-half years—and a "natural winnowing process"—for the show to feel like his own.