WASHINGTON—A critical part of the plumbing that keeps money flowing through the financial system is experiencing turmoil as new regulations prompt banks to step back from the multitrillion-dollar "repo" market.

The large and opaque market for repurchase agreements helps keep finance and trading moving, allowing hedge funds, investment banks and other financial firms to borrow and lend short-term funds, often overnight.

...