Defaults are rising in a key corner of the commercial real-estate debt market just as borrowing costs are set to jump, raising the likelihood of a slowdown of the $11 trillion U.S. commercial property sector in 2017.

A financial crisis-era regulation is about to take effect that is expected to make some commercial real-estate borrowing more expensive and complicated, analysts said.

At...