After spending five hours with ABC News’ George Stephanopoulos, it was hard to imagine that former FBI director James Comey had that much more to say. But that didn’t stop him from sitting down with Stephen Colbert for his first late-night interview on Tuesday following the release of his new best-selling book, A Higher Loyalty.

In that previous sit-down on ABC, Comey revealed that after he was fired by President Trump, he drank pinot noir from a paper coffee cup on his flight home from Los Angeles. So that’s what Colbert brought out during their chat in an attempt to loosen Comey up like he did with Jennifer Lawrence earlier this year. They toasted “to the truth.”

Comey told Colbert that he decided to write his memoir because he thought he “could be useful by offering people a vision of what ethical leadership looks like, and show people through a series of stories how ethical leaders make hard decisions,” adding, “it would be particularly useful now when our president is not that.”

With that in mind, Colbert asked if Trump still has a chance to turn things around and become an “ethical leader” in Comey’s eye. “I think it would be very hard given the way he is as a person,” Comey said. “He’s somebody who doesn't seem to be somebody who has external reference points his life.” By contrast, Trump is always asking himself, “What will fill the hole in me and get me the affirmation I need?”

Working from Comey’s theory that working for Trump was like working for a “mob boss,” Colbert asked, “Were you surprised that you got whacked? Because that’s what they do.”

Comey insisted that he “actually was surprised,” saying, “That would be a crazy thing to do. Why would you fire the FBI director who’s leading the Russia investigation?”

“Because you’re leading the Russia investigation,” Colbert replied.

At one point, Colbert tried to get Comey to fire back at Trump for calling him, among other nasty nicknames, a “slimeball,” but he didn’t take the bait. “I’m like a breakup he can’t get over,” Comey said. “I'm out there living my best life. He wakes up in the morning and tweets at me.”

“My first reaction to those kinds of tweets was to shrug and say, ‘There he goes again,’” he continued. “But actually I caught myself and I said, ‘Wait a minute, if I’m shrugging, are the rest of the country shrugging? And does that mean we've become numb to this?’ It’s not OK for the president of the United States to say a private citizen should be in jail. It’s not normal. It’s not acceptable.”

But like Fox News’ Chris Wallace, who described Comey’s book as “bitchy,” Colbert wanted to know why the former FBI director included digs about Trump’s orange skin and bizarre hairstyle in what has been billed as a morally superior account of the facts. “Why did you include that?” he asked.

“Because I’m trying to be an author,” Comey said. “And I’m sitting there typing, and I can hear my editor saying, ‘Bring the reader with you, show the reader that room.’” He defended himself by pointing out that he also describes how he was struck by how skinny President Obama is. He insisted that he’s not trying to “make fun” of Obama or Trump, but rather just trying to “report” what he saw.

Later, Colbert played the clip of himself announcing to his live audience last May that Trump had fired Comey. At that time, they applauded wildly for his ouster from the FBI.

“Have you made everyone in the world mad at you?” Colbert asked. “I was a little shocked. I thought the audience would be shocked when I told them. But in fact they were overjoyed that even the man they don’t like, President Trump, had fired you, and I think because of what they perceived you had done to Hillary Clinton.”

Comey agreed, saying the reaction was a “hangover from the Clinton email nightmare” of 2016. “When that case began, I knew we were going to piss off at least half of partisans,” he said. “It never occurred to me we would piss off all of them.”

After a break, Colbert continued to press Comey on why he decided to go out, on his own, during the presidential election and accuse Clinton of “extreme carelessness” in her use of a private email server. Comey said he was just trying to close the investigation in the “least worst way.”

When Colbert questioned Comey’s “rationale” for going against Justice Department “norms” when he released a letter to Congress that he was reopening that investigation just days before the election, Comey pushed back again, repeated his famous “speak” versus “conceal” rhetorical construction.

“‘Conceal’ naturally has a pejorative to it,” Colbert said. “It’s ‘speak’ or standard discretion of the FBI. It’s not the same thing as concealing.” He added, “One is bad immediately, one is possibly bad later, which sounds like Pascal’s Wager to me. So you were weighing a certainty that it’s bad now with maybe people would be upset later.”

“I didn’t see it as a maybe,” Comey said of the likelihood that Trump and the right would view Clinton as an “illegitimate” president had it come out later that he had not disclosed a reopening of the email probe. “It makes me sick to my stomach to think we might have had an impact,” he said. “I hope and pray we didn’t. But I hope this doesn’t sound strange, it wouldn’t change the decision.”

Answering Comey’s hypothetical about traveling back in time to do it over again, Colbert said, “You get back in the time machine, go back another 70 years, and kill baby Hitler. But then you come back to October and you don’t release the emails.”

Finally, Colbert pressed Comey to explain his comments to Stephanopoulos about not believing impeachment is the right solution to getting rid of Trump. “That sounds a little bit like you’re blaming the voters for the predicament we’re in right now,” Colbert said.

Comey said he “mostly blames those who haven’t voted,” even though as FBI director, he sat the election out in 2016. He did, however, confirm that he will be voting in 2020.