Investors know bitcoin’s violent mood swings well. What they often don’t know is that unscrupulous traders, wielding purpose-built software, can be behind them.

Manipulation in cryptocurrencies is a growing concern for regulators—and even for some proponents of the digital coins. The Securities and Exchange Commission cited that risk in August when rejecting several bitcoin-based exchange-traded funds. The office of New York Attorney General Barbara D. Underwood highlighted the issue last month in a report warning that crypto...