PODGORICA, Montenegro—Russian nationalists were behind an alleged coup attempt in Montenegro that included plans to assassinate the pro-Western prime minister because of his government’s bid to join NATO, the Balkan country’s chief special prosecutor said Sunday.

Milivoje Katnic said the investigation leads to the conclusion that “nationalists from Russia” organized a criminal group that planned to break into the Montenegro Parliament on election day, kill Prime Minister Milo Djukanovic and bring a pro-Russian coalition to...