The housing market may be coming back, but a growing number of policy makers have expressed concerns in recent months that it’s still too hard to get a mortgage.

Federal Reserve governor Elizabeth Duke outlined some of these concerns and their causes in a speech last month. She was quick to note — as is anyone else who has sounded similar alarms — that she doesn’t want the market to return to the go-go days of 2005 or 2006 when anyone who could fog a mirror could get a loan. “But I also don’t think it would be a good idea to go back to the quite restrictive credit conditions of the early 1980s,” she said in the speech to mortgage bankers.

The drop in purchase mortgages — loans for buying a home rather than refinancing an existing loan — has been most pronounced among borrowers with low credit scores. Originations have dropped by 30% for borrowers with credit scores above 780 between 2007 and 2012, but they’ve dropped by 90% for borrowers with credit scores between 620 and 680.