Story highlights Source: Sterling says he'll sue NBA if he's punished

The longtime Clippers owner is banned from team operations, facilities

He also owes a $2.5 million fine

His lawyer has successfully sued the NFL before

Los Angeles Clippers owner Donald Sterling has sent a letter to the National Basketball Association telling the league he won't pay his $2.5 million fine and rejecting his lifetime ban, a source close to the situation said Friday.

The letter also threatens the NBA with a lawsuit if Sterling's punishment for making racist remarks, handed down by NBA Commissioner Adam Silver in April, is not rescinded, the source said.

"We reject your demand for payment," the letter says, according to Sports Illustrated

In an exclusive interview with CNN's "AC 360" this week, Sterling repeatedly denied he is a racist despite remarks made in a private conversation with his friend V. Stiviano that were released online.

JUST WATCHED Will Sterling fight to keep the Clippers? Replay More Videos ... MUST WATCH Will Sterling fight to keep the Clippers? 04:46

JUST WATCHED New Sterling recording released Replay More Videos ... MUST WATCH New Sterling recording released 03:24

JUST WATCHED Sterling: 'I made such a mistake' Replay More Videos ... MUST WATCH Sterling: 'I made such a mistake' 01:55

JUST WATCHED Sterling: Racism is not a big problem Replay More Videos ... MUST WATCH Sterling: Racism is not a big problem 01:36

JUST WATCHED Magic to Sterling: Can't win this battle Replay More Videos ... MUST WATCH Magic to Sterling: Can't win this battle 01:16

In the recording, which drew widespread condemnation from fans, players and the league after it appeared on TMZ, Sterling chastises Stiviano for posting pictures online of her posing with African-Americans, including NBA legend Earvin "Magic" Johnson. He tells her not to bring Johnson to Clippers games.

"Admire him, bring him here, feed him, f**k him, but don't put (Magic) on an Instagram for the world to have to see so they have to call me," he said.

"In your lousy f**ing Instagrams, you don't have to have yourself with -- walking with black people," he said during a different portion of the recording.

Sterling said he was baited into saying those remarks.

Silver and the league moved quickly, announcing the punishment three days after the recordings were released.

"I'm apologizing, and I'm asking for forgiveness," Sterling told Cooper. "Am I entitled to one mistake? After 35 years. I mean, I love my league, I love my partners. ... It's a terrible mistake, and I'll never do it again."

The league's 29 other owners have taken the initial steps to force Sterling to sell the franchise he has owned for 33 years, with an advisory committee meeting three times so far.

Sterling's estranged wife, Shelly, has said she wants to keep her 50% ownership in the team, which is through a family trust.

The NBA last week tapped Dick Parsons, a former chairman of Citigroup and Time Warner, to be the team's interim chief executive.

Sterling has hired Maxwell Blecher, an antitrust lawyer, to represent him with the NBA, Sports Illustrated and USA Today reported.

Blecher represented the Los Angeles Memorial Coliseum in its lawsuit victory against the NFL that allowed the Raiders to move to the city from Oakland without approval from other owners before the 1982 season. The Raiders returned in 1995.

According to a 1982 Washington Post article, Blecher also represented Sterling when the Clippers owner wanted to move the team from San Diego to Los Angeles. The NBA fined the franchise $25 million when he did so in 1984, but Sterling sued until the league reduced the fine, according to multiple media reports.

CNN reached out to Blecher and the NBA on Thursday night but didn't hear back immediately.

Blecher reportedly argued that Sterling was being denied due process rights and he hadn't violated the NBA constitution

The Clippers' turbulent season ended late Thursday in a 104-98 loss to Oklahoma City. The Thunder won the series four games to two.

At a post-game news conference, Clippers coach Doc Rivers said he didn't think the team could use the controversy surrounding their owner as an excuse for bowing out of the playoffs.

"Obviously, it's nothing that anybody wants to go through -- nothing I want to go through -- but overall, I thought our fan base, I thought the city, and I thought our guys really hung in there ... very proud of them," Rivers said. "None of us signed on for this, but this is what happened."

Clippers point guard Chris Paul was in no mood about Sterling after the game.

"That's the least of our worries right now is him. We just lost the damn series," he said. "That's the last thing on our mind. We gave him too much attention as it is."