Donald Sterling and his antitrust lawyer Maxwell M. Blecher have sent a letter to the NBA threatening to sue the league, while also refusing to pay the NBA’s $2.5 million fine for his recent racist comments.

The letter, first reported by Sports Illustrated and USA Today, claims that Sterling did nothing wrong and that “no punishment is warranted.” Sterling’s lawyer contends that the 80 year-old’s “due process rights” were violated when he was permanently banned after a brief four-day investigation. Blecher also insists Sterling did not violate any article of the NBA constitution.

NBA Commissioner Adam Silver fined Sterling $2.5 million on April 29 and issued him a lifetime ban for the remarks, which were secretly recorded by his mistress and released on TMZ’s website last month.

The Clippers, eliminated from the playoffs Thursday night, are currently being overseen by ex-Time Warner chairman Richard Parsons.

Brian Windhorst of ESPN wrote of the Sterling letter: “Donald Sterling can refuse to pay his $2.5M fine all he wants to make a statement, but (the) league will just withhold it from TV payout money.” NBA team owners are scheduled to meet soon to force a sale of the Los Angeles Clippers, although Sterling and his attorneys could wage a rather lengthy legal battle against the league.

The Clippers, sold to Donald Sterling in 1981, were recently valued at $575 million by Forbes magazine.