NEW YORK (CNNMoney.com) -- Some major financial firms are getting anxious about giving back the billions in U.S. government rescue funds they took hold of late last year. But it may not be that simple.

David Viniar, Goldman Sachs' (GS, Fortune 500) chief financial officer, made headlines Wednesday when he voiced that very sentiment to attendees of a Credit Suisse conference, saying it would be easier for the company to run its business if it could pay back the $10 billion in capital it received from the government last fall.

And in a statement to CNNMoney.com Thursday, Bank of New York Mellon (BK, Fortune 500), which received $3 billion last year, said it was looking to redeem the government's investment "as soon as is practical."

These comments come at a time when public distaste for the nation's banking industry is boiling over. Taxpayers are seething over the fact that major financial firms continue to pay lavish bonuses and make seemingly frivolous purchases after getting government aid.

Earlier this week, President Obama struck back, announcing that the government would limit executive pay for any institution that accepts government financial aid in the future from programs such as the Troubled Asset Relief Program, or TARP.

Fearing that lawmakers will make further demands, it's no surprise that Goldman may want to free itself from more regulatory influence.

But Goldman, or other banks for that matter, can't just simply write the government a new check for the money they received last year. And even if they could, it might not be a good idea.

Goldman Sachs and the other top banks that were recipients of the first round of TARP funding, including State Street (STT, Fortune 500), Citigroup (C, Fortune 500), Wells Fargo (WFC, Fortune 500) and Bank of America (BAC, Fortune 500), weren't exactly given a choice about signing up when the program was first announced in October.

Federal Reserve chief Ben Bernanke and Federal Deposit Insurance Corp. Chairman Sheila Bair defended the move at the time, saying it would help stabilize the shaky banking industry.

Credit market conditions may have improved somewhat since then, but it remains to be seen whether Goldman Sachs, or any other leading bank for that matter, is indeed healthy enough to shun government assistance and go it alone.

Mark Lane, an equity research analyst who tracks Goldman Sachs and rival Morgan Stanley (MS, Fortune 500), which also received $10 billion in TARP funds, at Chicago-based investment firm William Blair & Co., is one disbeliever.

"[Goldman] cannot fund their business on an unsecured basis," he said "To say we don't need TARP funds doesn't make a lot of sense to me."

Last quarter's sharp downturn in the economy and dour capital markets activity rippled through the banking sector, resulting in billions of dollars in losses for many firms, including Goldman Sachs. Shares of most major banks have tumbled this year, as a result.

What's more, under the terms of the Treasury's capital purchase program, a bank can only buy out the government's stake as long as the money comes from an equity offering of a similar amount that meets government approval.

And in these market conditions, it seems highly unlikely that banks would be able to raise that much capital.

So many investors have gambled on U.S. financial firms over the past year and a half, only to watch those shares suffer another steep decline just months, or even weeks, later. In addition, an offering of new shares would cause substantial dilution to the value of the holdings of existing shareholders, something that banks may not want to risk.

Ultimately Goldman and other banks simply don't like being susceptible to the government's bidding, according to one hedge fund manager, especially as the White House gears up to unveil its latest efforts to deal with the financial crisis on Monday.

"What it really comes down to is Goldman Sachs does not have control of its own destiny," said James Ellman, head of San Francisco-based Seacliff Capital, a hedge fund specializing in financial services. "The President of the United States does and the President of the United States will tell us in 3 to 4 days what Goldman Sachs' options are."