WASHINGTON—Investors are frantically trying to learn everything they can about bitcoin—and so are regulators.

Furious trading in cryptocurrencies is testing many in the Trump administration who are eager to embrace financial innovation, after nearly a decade of tighter clamps on risk-taking put in place after the 2008 financial crisis.

The...