BEIJING—China is moving to rein in its fast-growing and innovative online finance sector, which offers a glimpse of how the rest of the world may someday handle money but has seen a number of high-profile abuses. ​​

Chinese Internet companies have transformed the average smartphone into a platform for cashless transactions, bank transfers, loans and investments far beyond what is common in the U.S. With many skipping credit cards entirely, Chinese people buy money-market funds, split a restaurant check and pay for services ranging from taxis to takeout food all within the same phone app.

Last year, nearly a quarter of China’s population—a number bigger than the total U.S. population—made payments online. Data provider Euromonitor International estimates that China’s mobile payments this year will total $213 billion, compared with $163.5 billion for the U.S. The biggest payment service, an affiliate of e-commerce giant Alibaba Group Holding Ltd. called Alipay, has 400 million users.

“These numbers are phenomenal when compared to other payment providers that operate globally,” said Ng Zhi Ying of Forrester Research Inc.

Internet companies are pushing to allow users to open online-only bank accounts on their smartphones, with their identities confirmed via selfies. Meanwhile, China’s peer-to-peer, or P2P, lenders, which connect investors with borrowers through online networks, are set to make $33.2 billion in loans this year, 43% more than in the U.S., and could see issuance triple over the next two years, according to Morgan Stanley projections.