BAGHDAD—Iraqi security forces launched a counteroffensive Monday to retake a province bordering the capital from Islamic State militants, in a major test of efforts to bring the country’s diverse and fractured fighters together on a common front.

The joint operation in Anbar province includes the army, federal police, Shiite militias and Sunni tribal fighters. Unlike the counteroffensive to retake the mostly empty city of Tikrit in March, which was dominated by Shiite militias, this force relies more heavily on Sunni tribal...