WASHINGTON—A startup best known for its popular encrypted messaging application is testing the limits of the U.S. government’s crackdown on digital assets, with both sides readying for a court clash in the biggest cryptocurrency case the Securities and Exchange Commission has levied.

Telegram is feuding with the SEC over its $1.7 billion sale of cryptocurrency in 2018. The agency is accusing Telegram of violating investor-protection laws, saying Telegram’s digital coin is actually a security and not a currency. The case, playing...