The world’s biggest social-media companies, under fire for failing to police content on their sites, have invited an array of outside groups to help them figure out who should be banned and what’s considered unacceptable.

That solution is creating a new set of problems—public fights, complaints and legal battles.

Silicon Valley giants Facebook...