Republican Rep. Darrell Issa declared Thursday that the embattled IRS official who refused to testify Wednesday had no right to do so, and is now looking to haul her back before his committee.

The chairman of the House oversight committee made the call after consulting with attorneys about IRS official Lois Lerner's bizarre appearance before the panel on Wednesday. Lerner, the head of the exempt organizations division which oversaw the controversial targeting of conservative groups, caused confusion Wednesday morning when she pleaded the Fifth and refused to answer questions -- but also delivered an opening statement in which she asserted her innocence.

Though Issa dismissed her from the hearing room, he questioned at the time whether she had waived her rights by delivering the statement. A spokesman told Fox News on Thursday that Issa had reached a decision.

"After consulting with counsel, Chairman Issa has concluded that Ms. Lerner's 5th amendment assertion is no longer valid," spokesman Ali Ahmad said. "She remains under subpoena, the Committee is looking at recalling her for testimony."

Issa, citing the concerns over Lerner's comments, never actually adjourned the hearing -- where other current and former Treasury and IRS officials testified. He only called it into recess. The thinking among Republicans is that they can still call her back to testify.

More On This... IRS official on leave refused to resign, GOP senator says

Issa told Fox News on Thursday that she can't put the genie "back in the bottle."

Republicans are accusing Lerner of trying to have it both ways.

"If you could do it the way she wants to do it, then every defendant would come, say 'I didn't rob the bank, and I'm not going to answer the prosecutor's questions,'" Rep. Trey Gowdy, R-S.C., told Fox News. "So we'd all in life like to get out our version without having to answer anyone else's questions. It's just not fair. And I don't think it's legal."

In her opening statement, Lerner asserted her innocence.

"I have not done anything wrong," she said. "I have not broken any laws. I have not violated any IRS rules or regulations, and I have not provided false information to this or any other congressional committee."

Lerner is represented by lawyer William W. Taylor, who is noted for winning a dismissal of all charges against former IMF director Dominique Strauss-Kahn in a high-profile sexual assault case.

It's unclear whether Lerner can avoid another round of questioning by the committee.

Rep. Elijah Cummings, D-Md., the top Democrat on the committee who was as tough as any Republican on the IRS witnesses Wednesday, said he thought Lerner was still in her right to refuse to answer questions.

"I'd like to see (hearings) run like a federal court. Unfortunately, this is not a federal court and she does have a right," Cummings said Wednesday. "And we have to adhere to that."

A brief assessment by the Cato Institute agreed, saying that unlike in a courtroom, Lerner could in a hearing room "selectively" invoke her Fifth Amendment right.

Former IRS Commissioner Doug Shulman, who stayed to testify Wednesday, endured a tough round of questioning for the second day in a row. Lawmakers were visibly frustrated as he struggled to explain why he didn't notify Congress after learning of the practice last year.

But lawmakers are itching to question Lerner, having aired a string of complaints about her own failure to notify Congress.

Lerner touched off the public controversy when, at an American Bar Association conference this month, she apologized for the IRS' practice of targeting conservative organizations for additional scrutiny. It was the first time the agency acknowledged the practice.

She said she hadn't revealed the information sooner, because she was never asked. But just two days before the ABA conference, Lerner was specifically asked about the investigation.

Rep. Joseph Crowley, D-N.Y., who had asked her about it, later called her answer evasive.

"The bottom line is you cannot lie to Congress, and you cannot be evasive, you cannot try to mislead Congress," he said.

Fox News' Chad Pergram contributed to this report.