White House budget chief Mick Mulvaney said Thursday that the authority wielded by the Consumer Financial Protection Bureau (CFPB) chief "should frighten people" days after he assumed the role of acting director.

Mulvaney, who also serves as the director of Office of Management and Budget, said on Fox Business Network's "Lou Dobbs Tonight" that the structure of the CFPB is "fundamentally flawed" and that his directive as its administrator would be to "limit" the agency's effect on capitalism.

"The structure of the CFPB is just fundamentally flawed. Authority that I have now as the acting director really should frighten people," Mulvaney said.

"You can sit down in a room with three or four people, and say, well, let's go off and do this, and there is no accountability to Congress. I could set the budget pretty much without any input from Congress - in fact, without any input from Congress," he added.

"We're going to try and limit as much as we can what the CFPB does to sort of interfere with capitalism and with the financial services market."

Mulvaney's remarks came after he was asked about his former comments as a GOP congressman when he ripped the agency as lacking accountability and being "a joke."

Mulvaney said Thursday that lack of congressional oversight for the agency has led to the bureau wielding too much power.

"There is no accountability to the American taxpayers here and that's wrong," he said. "I'm hopeful that it will change."

The White House budget chief's remarks came days after a federal judge rejected a lawsuit from a top CFPB official who claimed she was the rightful director of the agency instead of Mulvaney, Trump's temporary pick.

The decision, from a Trump-appointed judge, at least temporarily ended a showdown between Mulvaney and CFPB deputy director Leandra English, who was elevated by former CFPB Director Richard Cordray to be the agency's acting director last week.

Mulvaney on Thursday swiped at Democrats such as Sen. Elizabeth Warren (Mass.) and other critics who supported English, dismissing the idea that the agency would have remained independent from the Trump administration.

"As I explained to the folks at the CFPB, this was always going to happen," Mulvaney said.

"If they thought -- if anybody was over there working, if anybody [that] supports the CFPB, Elizabeth Warren, thought that this agency was going to be independent from this president forever, that was never going to happen.