A quant-trading firm backed by hedge-fund investor Steven A. Cohen and venture-capital firm Andreessen Horowitz has run into early problems.

Quantopian gained attention by creating a network of 160,000 users who developed more than 600,000 trading algorithms, grabbing the early lead in an emerging area of do-it-yourself quant trading.

But...