Big Wall Street firms have the most bruised public reputations, but it's a collection of smaller banks that continues to plague the Treasury Department's bank bailout program.

The latest report from the agency shows that more than 120 institutions - nearly all of them small banks - have missed their scheduled quarterly dividend payments, which is more than a sixth of the banks that received federal aid during the financial crisis.

In addition, five banks that received capital injections from the controversial $700 billion Troubled Assets Relief Program have failed altogether, making it highly unlikely that taxpayers will recover the nearly $3 billion poured into those institutions.

The Treasury report showed that at the end of August, a record six banks each missed six dividend payments. Saigon National Bank in Southern California has missed seven.

The rising number of "deadbeat" banks, as they are known, has prompted calls for Treasury officials to take action to protect taxpayers' investment.

The bailout legislation gives Treasury the authority to appoint two members to the boards of banks that miss six or more dividend payments. But the agency has refrained from doing so.

In its report, Treasury stated that in weighing whether to exercise its option to appoint directors, it would "prioritize" institutions in part based on whether the government's investment in the bank exceeds $25 million.

That list includes AnchorBank of Wisconsin, which received $110 million, and Seacoast National Bank of Florida, which received $50 million.

"We are exploring a number of options on how to properly exercise our contractual rights so to best protect the interests of taxpayers," Treasury spokesman Mark Paustenbach said.

Administration officials are quick to point out that overall, the TARP program has fared far better than initial projections, and that the estimated cost of the program has continued to dwindle (the non-partisan Congressional Budget Office recently lowered the projected final cost to $66 billion).

They say that while the missed payments from an increasing number of community banks is a legitimate problem, the amount of taxpayer money at stake pales in comparison with the government investments in firms such as General Motors and insurance giant American International Group.

In addition, they note that taxpayers have already recovered three-quarters of the TARP funds invested in banks.

Others expect the problem of missed dividend repayments to continue to grow, as small banks continue to struggle with the lagging economy and troubled loan portfolios. Most large and healthy banks long ago repaid their aid and exited the bailout program.

"The number of institutions missing their dividends will tend to go higher," said Linus Wilson, a finance professor at the University of Louisiana at Lafayette who has monitored the government's aid efforts. "You have the really strong institutions leaving the program, and the not-so-strong ones tend to get worse."

During the crisis, Treasury pledged to give aid only to banks that were in relatively good shape but that needed help to weather the financial upheaval. Looking back, Wilson said, some of the government's investments "were really speculative at the time."

He cited CIT Group, which failed despite a $2.3 billion infusion of taxpayer money, and OneUnited, the Massachusetts-based bank that has missed six dividend repayments and has come under scrutiny for seeking help from U.S. Reps. Barney Frank (D-Mass.) and Maxine Waters (D-Calif.).

Wilson said the wave of missed payments also should stoke skepticism about the administration's plan to make more taxpayer money available to community banks through a proposed small business bill that the Senate is expected to take up this week. Among other things, the legislation would create a $30 billion fund that smaller banks could tap to extend loans to businesses.

Administration officials have insisted that weak or troubled banks would be excluded from the program and note that to participate, a bank's federal regulator must deem it viable.