ANKARA—In the months before Turkey’s failed coup, the country’s spy agency penetrated online chat rooms and decoded millions of secret messages but found no mention of the plot, senior Turkish intelligence officials said.

Agents checked tips pointing to a coup that led nowhere. Analysts studied the video sermons of an imam who the government alleges directed the plot and speculated if the color of his robes relayed secret orders to his followers.

...