Some of the U.S.’s biggest proprietary traders and investors are testing the waters for a bigger move into bitcoin, giving a potential boost to the fledgling virtual-currency industry.

While still cautious of becoming exposed to “cryptocurrencies,” some of the firms, which trade with their own money on their own behalf, say they see potential for big profits in trading bitcoin as more investors enter the market and financial-services firms use the currency to streamline transactions.

...