The Securities and Exchange Commission denied an application for the first exchange-traded fund that would track the price of bitcoin, a high-profile rejection for the digital currency and the Winklevoss brothers behind the proposal.

Cameron and Tyler Winklevoss and their Winklevoss Bitcoin Trust first applied for approval of the product nearly four years ago. The bitcoin industry had been hoping an exchange-traded product would make the currency attractive to Wall Street and retail investors. The denial sets that back, at...