KABUL—The killings of American soldiers by Afghan troops are turning into a "rapidly growing systemic threat" that could undermine the entire war effort, according to a classified military study.

The study by Jeffrey Bordin, a political and behavioral scientist working for the U.S. Army in Afghanistan, warns that the magnitude of the killings "may be unprecedented between 'allies' in modern history."

Based...