Facebook Inc. said it would pay $5.7 billion for just under 10% of Indian telecom operator Jio Platforms Ltd., a massive expansion of the social media giant’s commitment to a promising market where it has faced difficulties.

The deal, unveiled late Tuesday, is Facebook’s largest overseas investment and gives it the opportunity to bring its WhatsApp messaging service—which has more than 400 million users in India—into closer partnership with the mobile operator that upended India’s telecommunications industry with cut-rate...