Nearly half the investors surveyed said they planned to cut back on purchases of homes in the coming year; in a survey last August, just 30 percent said they planned to cut back. Only 20 percent of investors said they plan to increase purchases, compared with 39 percent who said they would last August.

All this could have a significant impact on the housing recovery.

(Read More: Reverse Mortgages Backfiring on Seniors)



"If the investors gets sidelined—along with first-time buyers who are already sidelined—this housing market falls apart quickly," says Mark Hanson, a California-based housing and mortgage analyst. Hanson points to still-high levels of negative equity, which has kept many homeowners stuck in place.

Connecticut-based Carrington Mortgage Holdings, a hedge fund that had been buying distressed homes, recently stopped.

"We think the market is a little bit too frothy," said Carrington's Rick Sharga in an interview last month. Home prices are now up 12 percent from a year ago nationally, according to CoreLogic, but have risen far more greatly in formerly distressed markets where investors originally focused their purchases.

"The general consensus right now is that the bargains are drying up when it comes to buying foreclosed properties," adds Sharga.

(Read More: Rising Rates Turn Investors From REITs)



That is largely due to a lack of distressed homes for sale. The number of foreclosure sales in the first quarter of this year fell 22 percent from a year ago, according to RealtyTrac, a real estate website. The number of short sales, when the home is sold for less than the value of the mortgage, also fell, as rising prices provided less incentive for banks to agree to such deals. Some claim banks are actually holding onto repossessed homes, waiting for prices to rise higher.

Investors accounted for 19 percent of home sales in April, according to the National Association of Realtors, down from 24 percent in all of 2012. Investors include individual buyers as well as large hedge funds, but the hedge funds have been getting much of the attention, credited with juicing prices in the hardest hit housing markets like Phoenix and Las Vegas. Their so-called REO-to-Rent strategy (Real Estate Owned-to-Rent) has evolved into a new asset class, with two of the companies that engage in the practice going public this year as real estate investment trusts (REITs).