Billionaire Peter Thiel’s venture-capital firm is backing a startup that aims to bring Wall Street electronic-trading expertise to the bitcoin market—a bet that cryptocurrencies will continue to gain mainstream popularity.

Mr. Thiel’s firm, Founders Fund, is among the investors in the early-stage startup, called Tagomi Systems Inc., people familiar with the situation said.

Tagomi...