(CNN) Bill Cosby does not plan to testify in his own defense during his upcoming assault trial because of what might happen on cross-examination, the comedian said in his first extensive interview in more than two years.

"What people want to say, want you to say, and how they maneuver -- yes, I do have lawyers to protect me, 'objections,' 'sustained' -- but I just don't want to sit there and have to figure out what I believe is a truthful answer as to whether or not I'm opening a can of something that ... my lawyers are scrambling," Cosby said.

The rare comments from Cosby aired Tuesday in a taped interview with CNN host Michael Smerconish on his radio program on the SiriusXM POTUS channel. It was the first time since late 2014 that Cosby had publicly addressed a raft of claims against him. More than 50 women have accused Cosby of misconduct, many alleging he drugged and assaulted them.

In the radio interview, Cosby said that the "piling on" of accusations was "an impressive way to get public opinion" to tilt against him.

"Are you telling me that they're all lying?" Smerconish asked.

"You know better than that," Cosby said.

"Meaning, you can't answer that question, or you won't answer that question?" Smerconish asked.

"I won't, and I cannot answer that," Cosby said. "It's really not fair, and you know that because all I have to do is say something similar to that and the next thing I know the postman is carrying a big bag, or whoever it is, it's saying 'defamation, defamation, defamation.'"

Cosby faces three counts of felony aggravated indecent assault in connection with a 2004 case involving Temple University employee Andrea Constand. He has pleaded not guilty to all charges. The trial is set to start June 5, with jury selection due to begin next week.

Speaking out for his fans

With the trial looming, Cosby said that he is not at liberty to discuss any allegations against him. Still, he said he wanted to speak out for his remaining fans.

"I decided I think it's time for me to do something so that the people who still have faith in me, people who are still wondering what I sound like, as opposed to the National Enquirer, which is very interesting reading when they write about me," he said.

Cosby several times brought up what he described as unfair coverage by the news media.

"There's an old show business saying, 'Look, it doesn't make any difference as long as they spell your name right,'" Cosby, 79, said. "Well, I want to raise my hand and say I want to really debate that strongly."

Cosby's publicist told CNN he set up the interview to let listeners hear the "passion and concern" in Cosby's voice and to remind them of who Cosby is and who he always has been.

"I want people to hear that and hear the strength in his voice," Andrew Wyatt said. "He has a strong mind, and he's capable."

In the interview, Cosby said that he has "never" lost the support of his wife. He also said that he is in good health but that he is "legally unsighted" due to issues linked to glaucoma.

Cosby generally refrained from commenting on his legal situation, though he did say he was "confused" by a judge's ruling that released a previously sealed deposition in a civil case with Constand . That deposition is central to the upcoming criminal trial against Cosby.

"I have an emotion about what the judge did, and I'm still very much confused about how that came about and caused whatever is happening today," Cosby said.

Smerconish, who practiced law for 10 years, said he disagreed with the judge's decision to release that deposition, and Cosby agreed.

"It's the way it was put out and the way many people saw it, and you just said it, and I think the safest way to put it is, I agree with you," Cosby said.

Wyatt did not discuss the interview plans with Cosby's legal counsel before it happened, he said.

In a separate interview that the publicist provided in advance to Smerconish, Cosby said he'd like to be remembered for "being the guy that they made give back all the things they rescinded." Smerconish asked him to expand on what he meant by that.

"Things were rescinded," Cosby said, "and I'd like to get those things back because the people who decided to make that decision then saw they'd made a mistake."

In addition, Cosby suggested he may want to return to performing depending on the outcome of the trial and how that impacts public opinion.

"I still feel that I have an awful lot to offer in terms of my writing, in terms of my performance," he said.

'Racism' a factor, daughter says

In statements released Monday, two of Cosby's daughters spoke out in defense of their father and challenged the media narrative of his fall from grace.

JUST WATCHED Cosby calls sex assault allegations nefarious Replay More Videos ... MUST WATCH Cosby calls sex assault allegations nefarious 02:26

"I strongly believe my father is innocent of the crimes alleged against him, and I believe that racism has played a big role in all aspects of this scandal," Ensa Cosby said.

In the radio interview, Smerconish asked Cosby whether he agreed that racism has played a role.

"Could be. Could be," Cosby replied. "I can't say anything, but there are certain things that I look at and I apply to the situation, and there are so many tentacles."

Smerconish pointed out that the accusers are both black and white, and Cosby responded saying that they were "motivated" by different factors.

"When you look at the power structure and when you look at individuals, there are some people that can very well be motivated by whether or not they're going to work or whether they might be able to get back at someone," Cosby said.

In closing, Cosby thanked Smerconish and Sirius XM.

"I just hope I'm not in trouble now, man," he said.