Patrick Byrne made a career of provoking Wall Street. In his two decades as chief executive of Overstock.com Inc., he tormented short sellers, picked fights with journalists and burned millions of dollars on cryptocurrency projects.

That career ended Thursday when Mr. Byrne announced his resignation in a discursive letter to shareholders that alluded to a romantic relationship with jailed Russian agent Maria Butina and covert activities on behalf of the U.S. government.

It...