Facebook Inc. is recruiting dozens of financial firms and online merchants to help launch a cryptocurrency-based payments system on the back of its gigantic social network.

The effort, should it succeed, threatens to upend the traditional, lucrative plumbing of e-commerce and would likely be the most mainstream application yet of cryptocurrency. It comes as the social-media giant is under intense pressure from regulators, users and shareholders to address privacy shortcomings.

At...