Rep. Darrell Issa, who is poised to become the top congressional watchdog challenging the Obama administration, said Sunday that Attorney General Eric Holder should either "stop hurting the administration or leave."

The pointed words came after Issa was asked on "Fox News Sunday" about Holder's handling of several hot-button issues, including the possibility of legal action against WikiLeaks founder Julian Assange. Holder has said that the Justice Department is conducting an investigation into the leak of thousands of diplomatic cables, pledging to hold accountable anybody found acting in violation of U.S. law.

Lawmakers, though, have pushed him to charge Assange under the Espionage Act. Issa, the incoming chairman of the House Oversight and Government Reform Committee, suggested the administration needs to move faster.

The California Republican said Sunday that if President Obama is not treating the unauthorized disclosure as terrorism, then Holder needs to go after the leak as a criminal matter. "Otherwise the world is laughing at this paper tiger we've become," he said.

"He isn't doing enough," Issa said. "He's hurting this administration. If you're hurting the administration, either stop hurting the administration or leave."

Issa also criticized Holder for not doing more to investigate ACORN's use of federal funding and punish members of the New Black Panther Party who were videotaped outside a Philadelphia polling station in 2008, one wielding a nightstick.

The latter case drew the attention of the U.S. Commission on Civil Rights, which voted last month to approve a report criticizing the Justice Department for its handling of the matter. A voter intimidation case was dropped in 2009 against all but one of the defendants. The Justice Department has denied wrongdoing, saying the charges were dropped because the facts in the case did not support intimidation claims.