It's coming in fits and starts, but the housing sector is inching its way back from its disastrous collapse. Just last week, the Census Bureau reported new home sales were up 6.4 percent from March to April. But a few recent trends show that even while home sales slowly rebound, the housing recovery isn't exactly about average Americans realizing the picket-fenced American dream.

1. Lots of people with lots of cash

Cash sales — that is, buying a home without a loan — accounted for almost 43 percent of all purchases in the first quarter of 2014, according to real estate information company RealtyTrac. That's up from just 19.1 percent in the first quarter of 2013, as shown in the blue line in the chart below.

Cash purchases account for more than 60 percent of homes sold in five Florida cities, including Miami and Sarasota, and more than half of sales in cities including New York and Detroit. That might sound insane to anyone who has painstakingly scraped together a 20-percent down payment — how can so many homes be bought without a mortgage?

nearly 43 percent of homes are now bought in all-cash sales

According to the National Association of Realtors, 90 percent of those cash-only sales go to international buyers. Florida is one prime case of foreign investors scooping up new homes as investments (and pushing up prices in the process). In addition, cash buyers often get to jump to the front of the line of potential buyers, pushing borrowers aside. And some people are reportedly pulling money from IRAs to buy homes.

Either way, homes are selling, meaning there's some activity in the housing market that can spur economic growth. That's good, but it only signals that the housing market is recovering, not that the average American consumer is.

2. First-time buyers aren't coming back

In part because of those cash buyers, first-time homebuyers are being crowded out. First-time homebuyers by and large tend not to be cash buyers (only 13 percent of cash buyers are also first-timers), and the share of first-time homebuyers has overall declined slightly over the last couple of years, according to data from the National Association of Realtors.

But it's not just that first-time homebuyers are being beat out by investors; simple demography, combined with a rough economy, are keeping people away from buying.

"you still don't have many first-time homebuyers. ... that's because household formation has been weak."

"You still don't have many first-time homebuyers," says Adolfo Laurenti, deputy chief economist at Mesirow Financial in Chicago, Illinois. "That's because household formation has been weak: people who for whatever reason are not moving out of Mom and Dad's basement, people who are not getting married."

With lots of cash buyers and few first-time homebuyers, that means something is amiss in the housing recovery, he adds.

"It's really a matter of fundamentals on the demand for housing that are not really showing up yet," says Laurenti. "So I think this is a valid concern."

3. People can't scrape down payments together

In its most recent quarterly report on realtor confidence, the National Association of Realtors said a major factor keeping buyers out of the market is that they just don't have the money for a down payment.

"[B]uyers generally have to make a downpayment of 20 percent or more to be approved for financing," the NAR said in its report. "Unfortunately, the typical renter's savings are substantially less than the hefty downpayment and closing costs."

That means many Americans who could probably make the monthly payments on a home just aren't able to get that far, because nervous lenders want to give money only to the surest bets around — people who can hand over a big share of the home's cost up front (if not all of it).