It started with a warning, proceeded to a walkout and drew to a close with IRS officials walling off the agency from congressional interrogators whose patience appears to be wearing thin.

Democratic and Republican lawmakers were visibly frustrated Wednesday at the third congressional hearing on the IRS' practice of singling out conservative groups. Members of Congress asked, again and again, how the program started and why nobody in the agency's upper echelons notified Congress despite knowing about it at least a year ago.

One lawmaker, Rep. Stephen Lynch, D-Mass., warned at the top of the hearing that if witnesses did not start to open up, a special prosecutor would be appointed.

"There will be hell to pay if that's the route that we (choose) to go down," Lynch said.

Within minutes, though, the star witness -- Lois Lerner, director of the IRS division in charge of the controversial program -- invoked the Fifth Amendment and refused to testify.

More On This...

Committee leaders briefly tried to get her to change her mind, but she was ultimately dismissed.

Following that, lawmakers grilled former IRS Commissioner Doug Shulman but largely got the same answers Senate lawmakers heard at a hearing Tuesday.

Shulman again said he would not take "personal responsibility" for the creation of a list that was used to subject Tea Party and other groups to excessive scrutiny. And he provided few new explanations for why he did not notify Congress about the program despite learning about it in May 2012.

"Come on Mr. Shulman ... help us help the taxpayers," Rep. Elijah Cummings, D-Md., top Democrat on the House Oversight and Government Reform Committee, told Shulman.

Shulman continued to claim that last year, he did not have all the facts on the IRS program and knew that an inspector general investigation was underway, and so he did not loop in Congress on the matter.

However, committee Chairman Darrell Issa, R-Calif., revealed that another IRS official told the committee on Tuesday that an "IRS internal investigation" concluded in May 2012 and found "essentially the same thing" as the inspector general.

Yet Shulman, when asked about that review, said only that he didn't understand why they were calling it an internal investigation.

The committee also heard Wednesday from Inspector General J. Russell George and Deputy Secretary of the Treasury Neal Wolin. Wolin said there's "no indication" the Treasury Department was involved in the program, a claim Issa called "astounding."

But lawmakers reserved their most aggressive questioning for Shulman -- after having lost Lerner as a potential witness.

At the start of the hearing, Lerner, invoked the Fifth Amendment as she declared she would not testify or answer questions on the advice of counsel.

"I will not answer any questions or testify about the subject matter of this committee's meeting," she said.

Her comments prompted confusion in the hearing room, as she delivered a brief opening statement in which she claimed she had done nothing wrong. Issa said that given her other remarks, she may have "effectively waived" her rights.

After further deliberation, she was dismissed.

It was an unusual start to Congress' third hearing on the controversial IRS program.

Lerner, before invoking her Fifth Amendment right, said in a brief opening statement that she had done nothing wrong, even though people might assume that by her decision not to answer questions from the committee.

"I have not done anything wrong," she said. "I have not broken any laws. I have not violated any IRS rules or regulations, and I have not provided false information to this or any other congressional committee."

Lerner has emerged as a key figure in the scandal. She was the first to acknowledge the practice two weeks ago, and was aware of the program for years as head of the division. Though two officials have so far left the IRS in the wake of the controversy, she has not.

Lerner's attorney William Taylor III made clear in advance of the hearing she would not answer questions. He also asked Issa in a letter if she could skip Wednesday's hearing since she would be pleading the Fifth.

Taylor argued that forcing Lerner to appear "would have no purpose other than to embarrass or burden her."

Late Tuesday, the House oversight committee released a statement saying Lerner was still under subpoena and would be required to appear.