Nasdaq OMX Group Inc. NDAQ, -1.59% is testing a new use of the technology that underpins the digital currency bitcoin, in a bid to transform the trading of shares in private companies.

The experiment joins a slew of financial-industry forays into bitcoin-related technology. If the effort is deemed successful, Nasdaq wants to use so-called blockchain technology in its stock market, one of the world’s largest, and potentially shake up systems that have facilitated the trading of financial assets for decades.