Charles Oakley tells The Dan Le Batard Show he would decline if James Dolan invited him back to Madison Square Garden and won't comment on his ban being lifted. (1:43)

The New York Knicks lifted their ban on former player Charles Oakley on Tuesday, sources told ESPN, but Oakley says he just wants an apology.

"It's not about being at the Garden," Oakley told ESPN's Jeff Goodman. "It's about the fans. I want them to apologize to the fans. I told the commissioner I want them to apologize to the fans."

Oakley said he is "in pain now. I'm hurt."

Later on The Dan Le Batard Show on ESPN, Oakley was asked whether he would go back to Madison Square Garden as Dolan's guest.

"Right now, no," he said. "I told him yesterday."

"I have never asked for nothing," Oakley added. "I love the fans in New York. They've been supportive. One of the things I told the commissioner, I want to have a press conference and I want him to apologize to me and the fans. They've had my back and they've felt the pain. I really appreciate the people all around who've had my back."

Sources with both MSG and Oakley confirmed that the ban was lifted.

That news comes after NBA commissioner Adam Silver said he and Michael Jordan spoke with Oakley and Knicks owner James Dolan on Monday to help end the public dispute between Oakley, one of the franchise's most popular players, and Dolan.

Also, sources told ESPN that Oakley's camp and the Knicks had discussed having a possible "Charles Oakley Day" at Madison Square Garden, though that appears moot given Oakley's most recent comments.

Charles Oakley, center, was forcibly ejected from Madison Square Garden and arrested on Wednesday. ESPN

Oakley was forcibly removed from Madison Square Garden and subsequently arrested last Wednesday. After the arrest, Dolan announced that Oakley was banned indefinitely from the Knicks' home arena. Oakley was charged with three counts of misdemeanor assault and one count of misdemeanor trespassing. Dolan said the organization ejected Oakley because he was being verbally abusive to security while seated near the team owner. Oakley denied this in recent interviews.

After the incident, Dolan also suggested in multiple media interviews that Oakley has a drinking problem, which Oakley denied on Saturday in an interview with the New York Post.

Oakley and Dolan met with Silver at NBA headquarters, with Jordan participating by phone, the league said in a statement. It said both Dolan and Oakley apologized for the fallout from Wednesday's situation.

"Both Mr. Oakley and Mr. Dolan were apologetic about the incident and subsequent comments, and their negative impact on the Knicks organization and the NBA," Silver said.

Oakley said on ESPN that Dolan had apologized to him but added that "some things hurt. Some things just don't walk away. A dog who broke his leg isn't just going to walk away."

On Tuesday, Oakley told ESPN that he respects Jordan to the fullest and that he loves him, but he said: "I'm my own man."

The Knicks and Dolan have received criticism for the decision to ban Oakley and the public suggestion that he has a drinking problem.

Le Batard asked Oakley whether Dolan's comments hurt the most.

"That hurt my character more than the 10 guys that jumped on me," Oakley said.

Oakley didn't appear to care that the ban was lifted. He's worried about the impact of the situation on his character.

"If I was banned for the rest of my life it would have been cool. Can I walk down the streets of New York?" he said to Le Batard. "That's the most important thing."

Knicks coach Jeff Hornacek said the Oakley issue has been a distraction for his players.

"Yeah, for us, it is another thing this year that has been a minor distraction," he said. "Not a big deal for our guys, they handled it pretty well, but there is always talk about it. Oak was a great player for the Knicks and hopefully some day he comes back."

Carmelo Anthony said that the Oakley situation was a new one on him, but everything about the Knicks' season has been different, from his public spat with team president Phil Jackson to the constant trade rumors.

"I haven't seen anything like that, I don't think, ever," he said of Wednesday's scene. "The situation in the Garden, on top of kind of what I've dealing with the team. It's been a lot. I said this to you guys before, it takes special people to deal with. It's only here in New York. If you can make it here, you can make it anywhere. I believe that."

ESPN's Stephen A. Smith, Ian Begley and Ohm Youngmisuk, The Undefeated's Mike Wise, and The Associated Press contributed to this report.