Years into the economic and housing market recovery, financial woes still bar a significant number of Americans from achieving homeownership.

One in five middle-aged Americans who don’t own a home believe that they’re locked out of the market by credit scores that prevent them from being able to get a mortgage, according to a new Bankrate.com survey.

“There are some recession pains still lingering,” said Crissinda Ponder, a mortgage analyst for Bankrate.

Read More: Why Renters Are Waiting Longer to Take the Home Ownership Plunge

Other recent data suggest that relief could be on the way, but not all would-be homeowners will benefit equally. In a December forecast looking at the year ahead, TransUnion predicted that consumer lending, including mortgages, would be “fully recovered” by the end of 2016, although there are still about 7 million fewer mortgages today than there were in 2009. The credit bureau projected a year-end rate of “serious” mortgage delinquencies (those 60 days or further behind on payments) of roughly 2 percent, a nearly half-percentage-point drop from a year earlier.

Let our news meet your inbox. The news and stories that matters, delivered weekday mornings. This site is protected by recaptcha

If a recent Federal Reserve survey is any indication, the primary beneficiaries will be the people whose personal finances are already strong.

Last month’s Senior Loan Officer Opinion Survey found “a moderate easing of standards on some categories of residential mortgage loans,” the Fed said in its survey. “Banks, on net, indicated that they expected to ease standards on some categories of residential mortgage loans over 2016."

Additional data from the survey suggested that this is more likely to take place among prime and near-prime borrowers rather than those trying to obtain a mortgage through FHA loans, which typically cater to people with weaker financial profiles.

While 16 percent of all Bankrate survey respondents said they can’t qualify for a mortgage because of their poor credit, that rose to 20 percent among people between the ages of 30 and 64, suggesting that the long shadow of foreclosures is still lingering over many Americans' finances. As the seven-year period during which foreclosures (and most other negative activity) stays on a credit report comes to an end, the housing market could improve further, Ponder suggested, although she said it wouldn't likely to be a single, significant turnaround.

Read More: As Home Ownership Lags, Young Renters Left Behind in Wealth-Building

"I don’t think we’ll see the floodgates open up, but I think people will start trickling in as the foreclosures come off their credit," she said.

Another big stumbling block, especially for poor Americans, is saving up enough for a down payment, Bankrate found that among the more than 2,000 respondents to Bankrate’s survey, nearly three in 10 across all age brackets who don’t currently own a home say they can’t afford a down payment, and these percentages are higher among respondents who earn less than $30,000 a year or don’t have a college degree.

And even Americans with college degrees could be reluctant to enter the housing market.

Across all age brackets, 35 percent of the respondents to Bankrate's survey said they don’t own a home because they just don’t want to be homeowners yet; the prevalence of this answer was higher among adults under the age of 30 and college grads. Forty-four percent of young adults and almost half of college graduates gave this response, which could indicate that student loan debt repayment is one factor still weighing on the real estate market recovery, Ponder said.

"I just think it’s taking a while for people to get to a point where they feel like they’re ready," she said.

Read More: Home Ownership Dips to Lowest Level in Almost 50 Years