MAMASAPANO, Philippines—The military, with the help of the main Muslim rebel group here, is rooting out members of a splinter Islamist movement that is threatening a fragile peace process.

With military aircraft prowling overhead and soldiers scouring neighborhoods in rural Maguindanao, many residents feared the peace dividend they were promised last year was remote. Even so, the military sweep has the blessing of the Moro Islamic Liberation Front, the region’s largest rebel group, which, in a show of good faith has cleared...