Hedge-fund billionaire Robert Mercer, who gained attention outside Wall Street for supporting President Donald Trump, is resigning as co-CEO of Renaissance Technologies LLC.

For over two decades, Mr. Mercer helped Renaissance generate gains that few rivals could match, thanks to the firm’s pioneering use of closely held formulas and algorithms, an increasingly popular approach to trading known as “quantitative” investing.

But...