AYUTLA, Mexico—Masked men, rifles slung over their shoulders, stand guard on a lonely rural road, checking IDs and questioning travelers. They wear no uniforms, flash no badges, but they are the law here now.

A dozen villages in the area have risen up in armed revolt against local drug traffickers that have terrorized the region and a government that residents say is incapable of protecting them from organized crime.

The...