Falling volumes and spiraling costs are pushing trading firms out of U.S. options, raising concerns about fragility in a market that investors rely on to protect portfolios.

Trading has dwindled in most areas of the market, and investors and traders are grappling with increasing fragmentation.

Liquidity, the crucial ability to do trades without...