Some unusually large sell orders put bitcoin through a wild ride Sunday and Monday, highlighting what bitcoin advocates say is a need for advanced investing facilities to boost volumes and tame the extreme volatility in the digital currency’s thinly traded market.

Late afternoon in New York Monday, bitcoin was trading up 3.69% on the day at $331.45, according to Coindesk’s Bitcoin Price Index. In the early hours of Asian trading it had dropped to $301.29, in a brief revival of some of the furious selling seen on online markets...