Stephen Colbert was the featured guest on the season finale of Jerry Seinfeld’s popular web series Comedians in Cars Getting Coffee, during which the two jokesters cruised around Montclair, New Jersey, in a funky 1964 Morgan +4, stopped for lunch at Bluestone Coffee Group, and chatted about everything from a commercial Colbert was in before he was famous for the Nebraska bank FirsTier, to how they wished Steve McQueen were gay.

But the news that’s got everyone all abuzz? Colbert admitting that he was planning on leaving The Colbert Report even if he hadn’t gotten the gig replacing David Letterman as host of The Late Show on CBS. “I was ready to stop,” he told Seinfeld, “The Letterman thing fell in my lap.”

Colbert has been playing his character—a ridiculously conservative pundit in the vein of Bill O’Reilly—ever since he joined The Daily Show in 1997. That’s a long time to play one character, something Colbert pointed out to Seinfeld when he expressed surprise at Colbert being ready to quit.

“You’re one of the greatest television performers I have ever seen,” Seinfeld remarked to him. “What you did with that character, the energy and the fun, you never seemed tired.”

Nevertheless, Colbert said he was and this was not the first time Colbert had talked about wanting to leave The Colbert Report. In an interview he did with Judd Apatow for his new book Sick in the Head, Colbert said he was “sick of the model,” realizing, “If I want a change, I’ve got to leave.”

“I play a character on my show, and he’s modeled on punditry, and I no longer respect my model. That’s my problem,” Colbert told Apatow. “Regardless of whether I was moving on to something else after this show, I don’t know if I could have done it much longer, because you have to be invested in your model. And I really am not. I can’t watch that stuff anymore.”

And so on to prime time. CBS announced in April last year that Colbert would be replacing Letterman as the host of The Late Show, and he is set to start on September 8. Letterman hosted his final episode of The Late Show back in May, with a star-studded cast that included Seinfeld.

The Late Show should be change enough for Colbert, who will be playing it straight for once (no word on whether he’s keeping the beard), and though the format is not fully fleshed out yet, it is confirmed that he will continue interviewing celebrities the way Letterman did.

One thing Colbert is not interested in continuing? The rivalry between The Late Show and The Tonight Show on NBC, which was really more of a rivalry between the shows’ former hosts, Letterman and Jay Leno respectively. The two feuded over ratings and more ever since Leno was named successor to Johnny Carson instead of Letterman. The feud was fun to follow for a while (and even led to this Super Bowl ad with Oprah), but the joke thankfully retired with the shows’ hosts.

“I think nothing would be more boring than late-night war,” Colbert told Seinfeld, saying he even went out for drinks with the new host of The Tonight Show, Jimmy Fallon, right after his appointment to The Late Show was announced. “We laughed, told stories. That’s it,” he said “And it was lovely.”