The heat in housing demand is cooling across the nation and across most price points, but luxury is really feeling the chill.

High-end homebuilder Toll Brothers reported a sharp 13 percent annual drop in the number of signed contracts for homes in its fiscal fourth quarter. It also reported a 9.3 percent cancellation rate, up from 7.9 percent one year ago. Toll's average sale price in the quarter was $906,000 compared with the national average price of $294,200 in October, according to the National Association of Realtors.

Toll's CEO blamed the slowdown squarely on higher mortgage interest rates and on the fact that negative housing numbers were being widely reported.

"In November, we saw the market soften further, which we attribute to the cumulative impact of rising interest rates and the effect on buyer sentiment of well-publicized reports of a housing slowdown," CEO Douglas Yearley Jr. said in a statement.

He noted that the same thing happened in 2013, when the mortgage rates jumped due to the "taper tantrum" — the result of the Federal Reserve indicating it would reduce the amount of money it was funneling into the economy. Home sales fell when rates rose and only recovered, however, when interest rates fell back.

"We've been seeing a slowdown in housing all year, but this is the first time Toll Brothers really acknowledged it in a press release," said Peter Boockvar, chief investment officer with Bleakley Advisory Group. "I think up to this point they've felt like they're somewhat immune being on the upper end of the market where people are less sensitive to changes in mortgage rates. So that's what really stuck out to me is the acknowledgement of something we've known all year that it's now affecting their customer base."

The more recent housing reports to which Yearley alluded have come from every corner of the housing industry: homebuilders, Realtors, the U.S. Census and those who finance home loans. Most point to weaker affordability, which is now at a decade low according to several sources.