The virtual currency bitcoin suffered the biggest setback in its five-year history after a major exchange shut down on Tuesday, stoking concern about the future of a digital form of money traded by professional investors and ordinary people, but regulated by no one.

The abrupt closure of Tokyo-based Mt. Gox underscored the risks of a virtual currency that has seen a meteoric rise in the past year. Unlike a U.S. bank failure, in which deposits are insured by the government, there may be little recourse for people whose money...