James Dolan says anyone who acts like Charles Oakley did will get thrown out and banned from Madison Square Garden and that Oakley never should have made it to his seat. (1:56)

New York Knicks owner James Dolan said Friday that Charles Oakley has been banned from Madison Square Garden, two days after Oakley's arrest for getting into an altercation at a Knicks game.

"It's not necessarily a lifetime ban," Dolan said on The Michael Kay Show on ESPN New York 98.7 FM. "We need to keep the Garden a place that's comfortable and safe."

Oakley, 53, is a former Knicks player. He was arrested and charged with three counts of assault following an altercation near Dolan at MSG on Wednesday night.

The incident occurred during a stoppage in play as Kristaps Porzingis was shooting free throws in the first quarter against the visiting LA Clippers. A police source told ESPN that Oakley had purchased a seat a few rows behind Dolan and was directing comments at the Knicks owner, who eventually had security escort Oakley from the arena.

"Yes, I understand that he was a big star and a Knick, etc., but that doesn't excuse people from that kind of behavior," Dolan said.

"It is very clear to us that Charles Oakley came to the Garden with an agenda."