SHANGHAI—A $45 million corporate default in China this month is leaving 13,000 investors wondering whether they will be repaid and online powerhouse Ant Financial Services Group fielding uncomfortable questions about the quality of the investments on its platforms.

The fallout from a default by Cosun Group illustrates the complexity of business relationships in the country’s booming but largely unregulated financial-technology sector, which has channeled massive amounts of cash from individuals’ savings into loans to businesses.

...