MOSCOW—Russian authorities said Monday they had detained a U.S. citizen for alleged spying, setting the tone for more confrontation between the two countries in the new year.

American Paul Whelan was arrested on Dec. 28 while “carrying out spying activities,” Russia’s main security agency, the FSB, said in a brief statement without providing further details.

The...