The number of Americans who have cut the cord with mainstream banking has risen by 821,000 households in just two years, a newly released survey finds. One in 12 households or 8.3% are unbanked, while another roughly 20% are what researchers call "underbanked." We're ostensibly well into economic recovery, so why are more of us telling our banks, "It's not me, it's you?"

The number of Americans who have cut the cord with mainstream banking has risen by 821,000 households in just two years, a survey just released by the Federal Deposit Insurance Corporation (FDIC) finds. One in 12 households or 8.3% are unbanked, while another roughly 20% are what researchers call “underbanked,” meaning they have a bank account but still use payday loans, prepaid debit cards and other alternative financial products.

In 2009, the FDIC conducted a survey about Americans’ banking habits. At the time, the revelation that 7.7% of households were conducting their financial business without the benefit of a checking or savings account seemed surprising, but the country was just coming out of a deep and devastating recession at that point. In June 2011, when the survey released today was conducted, we were ostensibly 1½ years into economic recovery. So why are more of us telling our banks, “It’s not me, it’s you?”

The answer has a lot to do with fees. When financial reform knocked out a big chunk of fee revenue from automatic overdrafts and debit-card interchange, banks — especially big ones — realized they can’t afford to subsidize free checking the way they used to. The resulting monthly maintenance fees that have become increasingly common, along with fees for everything from getting copies of statements to closing an account, have prompted more people to ditch banks, either because they can’t afford the fees or are simply annoyed by them.

It’s this second group of consumers that banks should be worried about. An article in the Wall Street Journal profiles a family that earns $230,000 a year but shuns banks because the fees tick them off that much. While JPMorgan Chase & Co. executives famously told investors earlier this year that 70% of customers with less than $100,000 in combined deposits and investments could be unprofitable, fees might also drive away profitable customers like the affluent family profiled by the Journal.

On the other side of the spectrum are people who feel they’ve been priced out of the conventional banking system. Approximately one-third of Americans believe they don’t have enough money for a bank account, according to the survey; this was the most common reason people gave for not having a bank account.

(MORE: Study: Banks Still Doing a Crummy Job Disclosing Fees Even as They Raise Them)

The rise of the prepaid debit card from fringe product to checking-account substitute has facilitated the shift out of mainstream banking for a lot of people, both those who think they can’t afford an account as well as those who could pay the fees but don’t want to. According to the Journal, Green Dot, one of the biggest players in the prepaid-debit market, said its customers have an average income of $45,000, almost twice as much as it was before the recession in 2005.

As these cards have gained legitimacy, higher-income people view them as a cheaper alternative to checking accounts. Many of them are probably right. A recent study by the Pew Charitable Trusts found that prepaid cards are cheaper than checking accounts for many people. Most prepaid cards don’t levy $30-and-up overdraft penalties or have minimum-balance requirements, making them appealing to people who do a poor job of managing their finances. But Pew also found that even people who only go into the red or use an out-of-network ATM once in a while pay less in fees when using a prepaid card.

(MORE: Terrible Financial Advice: Top 10 Tips You Shouldn’t Follow)

Further blurring the line, mainstream financial institutions have begun offering prepaid debit cards. Consumer advocates worry this is just an attempt by banks to make an end run around regulations intended to protect consumers. In the conclusion of its prepaid card report, Pew urged the Consumer Financial Protection Bureau to take a look at these products and see if or how they should be regulated.

There are legitimate regulatory gaps that expose prepaid users to risk, but when people weigh hypothetical what-if scenarios against the tangible reality of how much it costs them to spend, save and borrow on a daily basis, it shouldn’t be surprising that more are opting for the choice they think will cost them less.