“The reaction on social media was critical and a lot of the dismay and anger was directed at me and my decision to engage him,” Mr. Remnick’s note said. “Some members of the staff, too, reached out to say that they objected to the invitation, particularly the forum of the festival.”

He added, “I don’t want well-meaning readers and staff members to think that I’ve ignored their concerns.”

Mr. Bannon lashed out at Mr. Remnick, calling him “gutless” for rescinding the invitation.

“The reason for my acceptance was simple: I would be facing one of the most fearless journalists of his generation,” Mr. Bannon said in a statement to The New York Times. “In what I would call a defining moment, David Remnick showed he was gutless when confronted by the howling online mob.”

Mr. Bannon, who is in Venice for the screening of a new documentary about him by Errol Morris, said that Mr. Remnick called him on Monday to say he could no longer accommodate him at the festival.

It was a swift and stunning turnabout for the festival, which is to take place in New York from Oct. 5 to 7.