Passions often run high at city council meetings in Oakland, California. But July 3, 2012 was something special. One after another, people stepped up to the microphone to plead with the council to stop paying its bills — and, for the most part, the council agreed.

Those bills were from Goldman Sachs.

The meeting came after months of public discussion about how to exit a 1998 deal — a fixed-rate swap agreement tied to debt issued to finance pensions — that wasn’t working out the way the city had envisioned.

The 5.6 percent fixed rate would be a good deal, the city figured, if interest rates rose to 8 or 9 percent in good fiscal times. But they didn’t. The market crashed. Interest rates hit the floor. The swap deal soured. Oakland was stuck paying $4 million in annual interest to Goldman Sachs. By that year, the swap had a negative market value of approximately $15.5 million, the city treasurer informed the council in the lead-up to the July hearing.

Since Oakland took out the first market-based bet on its pension fund nearly 30 years ago, municipal agencies across the country have used pension obligation bonds time and again to meet their bottom lines.

Municipal bonds used to be a city’s way of funding a big infrastructure project that would otherwise be beyond the reach of a general fund — almost everyone has needed a loan now and then. But as tax revenue fell and expenses rose, cities increasingly borrowed against payments they’d contractually promised to their own workers. When those deals didn’t pan out, some cities made even bigger bets, swapping their debts for variable interest rates offered by large financial institutions.

Many were losing bets. A handful of cities and counties went bankrupt balancing the debt on top of myriad other financial obligations. Others are stuck with huge interest payments on top of their unpaid principles.

Meanwhile banks continue to rake in the profits at the expense of taxpayers, despite federal inquiries and settlements.

Thanks mostly to a newly recovering market, American cities are limping back toward solvency, and the municipal bond market is making a comeback. But rebound aside, urban America remains sorely dependent on Wall Street and continues to pay dearly for the relationship — and it’s not clear how broke municipalities can get out of the bind.

Municipal bonds are one of the oldest kinds of formalized debt, dating back at least to the Italian Renaissance. America has always loved them, relying on bonds to develop infrastructure nationwide since the 1800s.

As local purse strings constrict ever tighter, cities have leaned even more heavily on debts to keep things running. Pension obligation bonds have proven to be one of the more popular lines of city credit. The premise of these bonds is that cities are able to postpone contributing to pension funds by borrowing from banks at a lower rate of interest than those invested funds will earn over the long term. As of 2009, Oregon, Illinois and Connecticut each have taken on more than 10 percent of their annual revenue on pension bonds.

“Pension obligation bonds were sold as a way to catch up quickly if you’ve taken big holidays in the past,” says Saqib Bhatti, a fellow at the Roosevelt Institute and director of the ReFund America Project.

When cities found their bottom lines still wanting, some gambled on interest rate swaps in an attempt to make up even more of the difference even more quickly. Essentially, they bet that the market where they’d invested their pension funds would continue to perform well, keeping interest rates high, so they traded their variable interest rates for fixed rates offered in new contracts by financial institutions that promised low risk and high savings.

The market crashed, and so did those invested pension funds. “So all the times that instead of putting the money toward the pension you put it somewhere else — now there was a huge shortfall,” says Bhatti.

Interest rate swaps only added insult to injury as the Fed lowered interest rates drastically while cities were still locked in to their old high interest rates on their bad bets with the banks.

“You couldn’t get out of one of these swaps unless you paid out all of the bank’s future potential earnings in one fell swoop,” says Turbeville. “You could be in a 25-year swap with 20 years remaining and if you wanted to get out you had to pay the bank the expected value over the length of the deal all at once. Fixed-rate bonds typically allow for refinance with a slight penalty — but with the swaps, you’ve lost that opportunity. The banks would just take it all.”

All those maxed-out credit cards and failed spins around the roulette wheel caught up with more than a few cities.

As of 2009, most pension obligation bonds were in the red, according to analysis by the Center for Retirement Research at Boston College, but their gains have risen along with the rest of the market in more recent years. The final outcome will depend on the market’s next moves.

“While the story is not yet over, since about 80 percent of the bonds issued since 1992 are still outstanding, some may end up being extremely costly for the governments that issued them,” the report’s lead author Alicia Munnell wrote with colleagues Thad Calabrese, Ashby Monk and Jean-Pierre Aubry.

A key takeaway of Munnell’s report is that these bonds can, and do, help governments defray pensions costs but, as she writes, that only happens “in the hands of the right government at the right time.”

Oakland provides a solid example of just how critical timing is. In 2012, the city treasurer concluded that the swap had yielded a net benefit of $37.5 million in present-value savings from the swap. Earnings would have likely continued to outpace losses if the market hadn’t crashed in 2007. Furthermore, the consultant hired by the city treasurer to assess the swap did not find any evidence that Goldman had overcharged taxpayers. The unforeseen financial crisis and an overabundance of trust in the market was to blame.

For some cities, the bad bets have yielded grave consequences.

In 2005, Stockton, California was riding high, approving public projects left and right, enjoying the boosted property tax revenue of a bubbling housing market. Only two years later, it was on the verge of collapse. The subprime mortgage crisis had gutted the city and it was leading the nation in foreclosures. The stream of tax revenue the city had gotten used to dried up almost overnight. It was a fiscal drought the city hadn’t planned for and quickly the coffers emptied. In short order, Stockton owed $152 million to the state pension fund.

In 2006, a Lehman Brothers representative came to town with an enticing offer: Make up that pension shortfall with $152 million in fixed-rate bonds. City officials weren’t sure they quite understood how the whole arrangement would work but they were desperate to keep parks open, police cars running and garbage men working; Lehman was offering a solution, even if nothing could be “guaranteed.”

The bet didn’t work and Stockton never made up that pension shortfall. In 2012, the city filed for bankruptcy. The judge in the case is still considering whether employee pensions can now be cut along with the city’s other debts, calling the situation “a festering sore.” Lehman Brothers, meanwhile, went bankrupt before Stockton.

Detroit’s collapse was not as sudden as Stockton’s — that once-great capital of American manufacturing had been in decline for decades. As the city shrank and its industry contracted, so too did its share of property and income tax revenues. Struggling to stay afloat as the state of Michigan also cut funds to the city, in 2005 and 2006, Detroit did as many others have done: it took out bonds in order to stay alive and avoid bankruptcy. Further, the city took out $800 million of those bonds on fixed rates that cost the city untold extra millions in inflated interest.

As Detroit inched toward bankruptcy, its bond rating dropped, triggering a clause common in interest rate swap deals, and giving Wall Street an opening to collect all its once and future fees. Ultimately the banks were paid less than they wanted. But the hit they took is dwarfed by the sacrifices made by Detroit taxpayers and its pensioners. The judge in Detroit’s bankruptcy case readily declared that pension debts were like any other debts, and could also be paid back at pennies on the dollar. The city’s retirees appear resigned to their fate, voting in July to cut their pensions and end most cost-of-living increases.

Cities may have been desperate, but they didn’t go looking for Wall Street. “There’s no question they were approached,” says Turbeville. “The investment banks and the commercial banks are constantly approaching municipalities with ideas — that’s how they compete.”

When Turbeville was at Goldman Sachs, he witnessed the competition for municipal clients firsthand. Oakland “was considered a wonderful client because they would do lots of exotic deals,” he says. “They did many, many very complex transactions during that whole period. They were at the very top of the stack in terms of the, shall we say, the quality of the client.”

Oakland is not Stockton, nor is it Detroit. By many accounts, the city is now on the rise — Oakland’s proximity to Silicon Valley’s booming tech industry has boosted its business and real estate markets, in turn filling city coffers with new tax revenue, its first surplus in years. By many accounts, it’s a “renaissance” for a city long-troubled by unemployment, crime and debt.

But the growth isn’t yet enough to push the city out of its fiscal hole. Oakland still faces $1.5 billion in long-term pension liabilities that it has promised to have under control by 2026. Some call it a “fiscal time bomb,” even with the city’s newly boosted income.

Meanwhile, Oakland’s swap with Goldman Sachs doesn’t expire until 2021. It continues to pay $4 million in interest each year.

Goldman Sachs declined to comment on the bank’s negotiations with Oakland. But at an annual shareholders meeting in May 2012, CEO Lloyd Blankfein was asked about efforts by Oakland to exit the 1998 interest-rate swap without paying a $14.8 million termination fee. He responded sharply.

“If we are at the lowest level of interest rates today, that means that (for) every fixed loan that happened before this, it would be advantageous to the borrower to tear that up and re-borrow today when interest rates are lower,” he said, according to CNBC. “That’s not how the financial system could work. … I don’t think it’s a fair thing to ask.”

Today, the city continues to pursue a settlement on the swap, as well as a debarment that would prevent Goldman from bidding on Oakland’s debt deals in the future. It’s a gutsy, expensive move that Oakland council member Dan Kalb said last year he would “reluctantly support”: “I’m so disgusted at Goldman Sachs and what they did to our community.”

Stockton, Detroit and Oakland may be salient examples of this crisis, but they are hardly alone. Hundreds of cities and counties are living with the reality of rate swaps gone wrong.

Only a few have been able to extricate themselves from these bad deals.

Even when cities are lucky enough to extricate themselves from financial doom, there’s still the matter of responsibility. “Sort of like predatory lending, the issue is whether it was ever possible for a city council to understand what they were getting into. Nobody wants to sort of suggest that they made mistakes in the past, especially if the mistakes are not just in one place but many places. That’s asking a lot of a public official to own up to the fact that they didn’t fully understand the transactions that they were assigned to do,” says Turbeville.

“You’re approaching borrowers who are in dire financial straits,” says Bhatti. “It’s not necessarily incompetence on the part of city officials. These products are relatively new, and the ones who really understand them are the ones who created them and are selling them. One of the things Wall Street firms have been very good at is when there’s a crisis, they find a way to use it.”

Not only did banks sell cities on these bad deals without fully describing the risks — a possible crime in itself — but some financial institutions also conspired to be sure they were bad, colluding in order to make it seem as though cities received multiple competitive bids when in fact they only really received one, inflating the lending institution’s rate of return even further.

But not all pension bond deals are bad, and not all rate swaps are rigged.

Given voters’ dual resistance to raising taxes and reforming public retirement systems, it’s likely that cities will depend on banks for debt issuance and bonds long into the future. These kinds of debts will continue to play a significant role in city finances.

Carrying debt is not an inherently bad municipal management strategy, and for all these problems, municipal bonds are back on the rise along with the rest of the market. But it’s probably in everyone’s best interest that the forecast not look too sunny.