NORRISTOWN, Pa. -- A judge refused to throw out the sexual assault case against Bill Cosby on Wednesday, sweeping aside claims that a previous district attorney had granted the comedian immunity from prosecution a decade ago.

Common Pleas Judge Steven O'Neill issued the ruling after a hard-fought two-day hearing.

Cosby's lawyers tell CBS News they will appeal.

Get Breaking News Delivered to Your Inbox

In another setback for the defense, the judge also denied a request to disqualify newly elected District Attorney Kevin Steele from the case. Cosby's lawyers had accused Steele of making a "political football" out of Cosby during the campaign.

The prosecutor in the sexual assault case against Bill Cosby argued Wednesday that a previous district attorney had no legal authority to make a deal a decade ago that supposedly shields the comedian from ever facing charges.

Montgomery County District Attorney Kevin Steele said that the no-prosecution agreement -- which was never put in writing and was only alluded to in a press release -- ran afoul of the law and was a misuse of authority.

"A secret agreement that permits a wealthy defendant to buy his way out of a criminal case isn't right," Steele said. "That isn't a commonwealth and prosecution stand."

Steele made the argument on Day 2 of a bid by Cosby's lawyers to get the sexual assault charges thrown out. They contend that then-District Attorney Bruce Castor's 2005 decision not to prosecute bars his successors from filing charges.

"In this case, the prosecution should be stopped in its tracks," Cosby lawyer Chris Tayback argued. "Really what we're talking about here is honoring a commitment."

Comedian and actor Bill Cosby depicted in a Feb. 3, 2016, sketch, drawn in a Norristown, Pa., courtroom. JANE ROSENBERG

Cosby, 78, was arrested and charged in December with drugging and sexually violating former Temple University athletic department employee Andrea Constand at his suburban Philadelphia home in 2004. The former TV star could get up to 10 years in prison if convicted.

Castor found the case too weak to prosecute in 2005, but Steele's office reopened the investigation last summer, after the comedian's damaging, decade-old testimony from Constand's civil case was unsealed and dozens of other women came forward to accuse Cosby of assaulting them.

Damaging testimony from Constand's lawsuit was unsealed last summer, prompting Castor's successors to reopen the case and ultimately charge Cosby.

Cosby admitted in the deposition that he had affairs with young models and actresses, that he obtained quaaludes to give to women he wanted to have sex with and that he gave Constand three pills at his home. He said he reached into her pants in what he insisted was consensual contact.

Comedian and actor Bill Cosby depicted in a Feb. 2, 2016, sketch, drawn in a Norristown, Pa., courtroom, where Cosby's attorneys asked a judge to dismiss a criminal sexual assault case. Jane Rosenberg

On Tuesday, Castor testified that as an elected representative of the state, he had the power to give Cosby a lifetime pass from prosecution.

He said he wanted the agreement to force Cosby to testify in Constand's civil case without invoking his Fifth Amendment right against self-incrimination. The strategy worked: Cosby testified, and Constand settled for an undisclosed amount.

Cosby's lawyers said they never would have let him testify if they didn't believe criminal charges were off the table.

However, no such agreement was ever put in writing in a legal document. Because of that, Cosby's lawyers have relied heavily on Castor's interpretation of whether the deal blocks prosecutors from revisiting the case.

"There is no legal authority allowing a district attorney to act unilaterally and confer transactional immunity on a defendant," Steele said. "We have found nothing to support this sovereign edict theory that was presented yesterday."

Actor and comedian Bill Cosby arrives for his arraignment on sexual assault charges at the Montgomery County Courthouse in Elkins Park, Pennsylvania December 30, 2015. REUTERS

The judge said he struggled to find similar cases where a suspect who was never charged received a promise that he'd never be prosecuted. Normally, immunity is granted after a suspect is charged because he or she can provide testimony or information to prosecutors.

Here, Castor said he was clearing the way for Cosby to testify in a lawsuit that, at the time of his decision, hadn't been filed yet.

"There is one thing we can all agree on," Tayback said. "There is no case quite like this."

Dozens of women have accused Cosby of drugging and sexually assaulting them since the 1960s, destroying his good-guy image as America's Dad. But the statute of limitations for prosecuting him has run out in nearly every instance. This is the only case in which he has been charged.

Castor defended his decision not to bring charges, citing among other things Constand's yearlong delay in reporting the allegations, her continued contact with Cosby, and suggestions that she and her mother might have tried to extort the TV star.