KABUL—Afghanistan’s security forces are fraying as Islamic extremists gain control over more territory, a U.S. government watchdog said, raising questions about progress in the yearslong project to make Afghans responsible for their country’s defense.

Afghan military and police forces, the cornerstone of American strategy to stabilize the country and prevent it from becoming a staging ground for terrorist attacks abroad, have shrunk by about 10% in the past year, according to a report released Tuesday by John F. Sopko,...