NEW YORK (CNNMoney.com) -- Much of the money given to General Motors and Chrysler to prevent them from collapsing will never be recovered, according to a report released Wednesday by the Congressional Oversight Panel.

"Although taxpayers may recover some portion of their investment in Chrysler and GM, it is unlikely they will recover the entire amount," the report says, citing estimates from the Treasury Department and Congressional Budget Office.

The oversight panel, headed by Harvard University professor Elizabeth Warren, was created by Congress last year to oversee the $700 billion Troubled Asset Relief Program.

GM and Chrysler were each teetering on collapse this spring when the Obama administration effectively forced both automakers into bankruptcy, lending them enough to survive. Both have shed billions of dollars in debt and are now rebuilding.

All told, since late last year, the government has provided or pledged the two companies, icons of American manufacturing, more than $60 billion in aid.

Treasury estimates that about $23 billion of initial loans to the two companies "will be subject to 'much lower recoveries,' " the panel's report says. In particular, $5.4 billion of loans to Chrysler are "highly unlikely to be recovered," it continued.

"The initial loans made last fall as the industry was imploding and when no restructuring plan was in place are not likely to be repaid in full," Warren said during a conference call with reporters.

How much of the remaining funds will be recovered is impossible to predict, Warren said, because the loans have been converted to stock.

"The American taxpayer is now an equity investor in Chrysler and GM," Warren said. "And the return on its investment depends on what those companies are worth in a year or two."

The government owns 10% of Chrysler and 61% of GM.

Rep. Jeb Hensarling, R-Texas, who is the only member of Congress on the oversight panel, declined to sign the final report. In a statement, Hensarling cited his objections to the use of TARP funds to help the auto industry and the structured bankruptcies of the two automakers.

"By making such an unprecedented investment in Chrysler and GM the Administration by definition chose not to assist other Americans that are in need," Hensarling said in a statement. "The government clearly picked winners and losers."

'Somewhat mixed' record

On the whole, the report said Treasury had a "somewhat mixed" record in how it handled the two bailouts.

Officials acted "aggressively" and demanded concessions from the companies. Yet they were not fully clear with the public about "the decisions to enter into the transactions in the first place."

The report made several recommendations. One was that the government's shares in GM and Chrysler should be placed in an independently managed trust to prevent any potential political entanglements, the report said. That would be preferable to simply holding the shares as a "passive" investor.

Warren said the question of whether TARP gave Treasury authority to help the auto industry "is the subject of considerable debate," the report said.

Treasury should provide a detailed legal analysis of this use of TARP funds, Warren said. Still, she said, there is unlikely to be a serious legal challenge to Treasury's actions in this case.