While his funds were closed to outside investors from 1995 to 2009, Mr. Greenblatt devoted time to raising five children, began teaching a value-investing course at Columbia Business School and became a prominent backer of charter schools in New York City.

In 2003, with his partner Robert Goldstein at Gotham, he started what became a $35 million research effort that first produced “The Little Book” and later the current fund series. “The idea of the project was: Could we prove what I had been teaching students and we had been using ourselves actually works on a broader scale?” Mr. Greenblatt said.

The stock-picking formula in the “Little Book,” which sold 300,000 copies and led to more than 30 other “Little Book” titles by the same publisher (“The Little Book of Currency Trading,” “The Little Book of Big Dividends”), relied on statistical measurements of companies’ pretax, pre-interest earnings yield and return on capital.

Four years later, Mr. Greenblatt decided to reopen his doors and take outside money again under an updated system that aims to customize the results of value screens similar to the “magic formula.” Every quarter, 13 Gotham analysts feed 2,000 companies’ earnings reports into a database, making adjustments for factors like pension obligations and legal claims. Gotham then picks stocks from the top 20 percent of the field and shorts stocks from the bottom.

Turnover is high, up to 399 percent a year, a possible pitfall. As of the funds’ last reports in July, the largest holdings included Gilead Sciences, a California biotech company, and Delta Air Lines. Its shorts were led by Medidata Solutions, a health care software provider, and Salix Pharmaceuticals. But Mr. Greenblatt notes that the screens do not exclude prominent stocks like Apple and Google, which he said on CNBC in mid-2013 were cash-generating “bargains hiding in plain sight.”

The liquid alternatives market, which includes hedge funds, managed futures and other exotic offerings, has mushroomed to $161 billion from $38 billion in 2008, by Morningstar’s count. Facing customer demands for low-fee index funds, Wall Street has been selling “liquid alts” as a lower-volatility alternative to traditional stock and bond funds at a time when both markets are considered richly priced.

But the Gotham funds, like other alternatives, come with risks and high fees. The largest, the $2.8 billion Gotham Absolute Return Fund, which was started in August 2012, owns stocks equal to 120 percent of its assets and has “short” bets on price declines in stocks equal to 60 percent of assets. While the offsetting longs and shorts leave a net market exposure of about 60 percent of assets, the combined value of the positions, at 180 percent of assets, means more that could go wrong.