WASHINGTON (MarketWatch) -- The government has spent trillions to rescue the banking system from the brink of disaster, but the biggest cost may be the loss in the government's credibility from an understandably distrustful and angry public, according to a quarterly report released Wednesday by the congressional watchdog over the bank bailout.

"The anger, cynicism and distrust created must be chalked up as one of the substantial, albeit unnecessary costs" of the Troubled Asset Relief Program, said Neil Barofsky, the special inspector general for the TARP, in a report to Congress. Read the report.

Barofsky argued that the Treasury and other agencies haven't done nearly enough to be open to the public about what they are doing and why.

Other critics have an even-harsher appraisal.

"This administration has not gone anywhere close to far enough to reform the banking sector," said former New York Gov. Eliot Spitzer, in an interview on CNN. "Tim Geithner had all the negotiating power in the world; he didn't use it. Tim Geithner continues to be the voice of Wall Street, not Main Street," Spitzer said, referring to man who is now Treasury secretary and served as president of the New York Fed during last fall's crisis.

In his report, Barofsky said the $699 billion TARP program had accomplished much of what it was intended to do when it was approved by Congress last October. Of the $699 billion, $317 billion is still available. The administration announced plans Wednesday to spend some of that money to boost lending to small businesses through community banks.

"There are significant signs of improvement in the stability of the financial system," he said, adding that TARP and the other measures by federal agencies to shore up the foundations of the system "played a significant role in bringing the system back from the brink of collapse."

Not all is well in the financial system or in the economy, he said, noting risks of increased foreclosures, rising unemployment and the threat from bad loans in the commercial real estate sector.

However, the successes must be weighed against the costs, which Barofsky said would be "substantial" in three broad categories. The direct costs that can be seen on the government's balance sheet, and two intangible costs that cannot be measured directly: the increase in moral hazard and the blow to the government's credibility.

Barofsky spoke as congressional committees continued work on legislation that would reform and repair the way the financial system is regulated, in an attempt to make sure the financial crisis of 2007, 2008 and 2009 cannot be repeated. See full story.

On the direct costs, Barofsky said "it is extremely unlikely that the taxpayers will see a full return on their TARP investments." Although some of the banks have repaid the money they were given, with interest, some of the TARP programs, such as mortgage modification or the investments in General Motors and Chryslers and in American International Group Inc. will never be repaid in full.

The costs of moral hazard have yet to be tallied. Much depends on whether regulatory reform legislation will find a way to eliminate the explicit taxpayer guarantee that big banks now enjoy. "The firms that were 'too big to fail' last October are in many cases bigger still, many as a result of government-supported and -sponsored mergers," the report said. ("Moral hazard" refers to the problem that any guarantee or insurance is bound to increase risky behavior, because the costs of failure will be paid by someone else.)

"Absent meaningful regulatory reform, TARP runs the risk of merely re-animating markets that had collapsed under the weight of reckless behavior," Barofsky said. Reform of the rating agencies is particularly crucial.

Barofsky reserved his strongest language for the third cost, the public's growing distrust of government. He said government's ability to act in crises depends on its credibility with the market and with the public, whose political support was crucial.

"Unfortunately, several decisions by the Treasury ... have served only to damage the government's credibility and thus the long-term effectiveness of TARP," he said.

The belief that "Treasury is just too closely aligned with the interests of Wall Street are only reinforced by Treasury's failures of transparency," Barofsky said.