Last year was a rough one for actively managed mutual funds.

Sure, the stock market saw double-digit growth again, and bonds had a surprisingly strong year. But now the taxman cometh, and many mutual fund investors with taxable accounts are finding out that their actively managed funds realized significant capital gains, which they must now pay taxes on. (When a mutual fund sells some of its holdings at a gain, it must pass through the taxable income to shareholders - even if those distributions are then reinvested in the fund.)



Some 453 mutual funds estimated their capital gains distributions for 2014 would be more than 10 percent of net asset value, 52 estimated distributions would exceed 20 percent, and 12 projected distributions of 30 percent or more as of Dec. 15, according to CapGainsValet.com, a website that tracks mutual fund distributions. The site's creator, Mark Wilson, chief investment officer of The Tarbox Group, a California-based wealth management firm, said that by his estimate, "maybe two or three index funds had greater than 10 percent distributions." The other funds were all actively managed.

The larger distributions, and resulting tax bills, from the actively managed funds would be easier to swallow if they were accompanied by extraordinary performance. But a majority of actively managed mutual funds are being outperformed by passive funds, including exchange-traded funds (or ETFs), that mirror popular indexes. About 85 percent of active large-cap stock funds failed to beat or even match their benchmark index through Nov. 25, according to Lipper, a research unit of Thomson Reuters.

Read More Holiday spending hangover? Get your finances back on track

Not surprisingly, investors are wondering: Does it make sense anymore to invest in traditional actively managed mutual funds?

"In taxable accounts, actively managed funds are really tough," Wilson said. "It's really difficult to beat indexing."

If recent inflow and outflow patterns are any indication, a growing number of investors agree.

In December, fund-research form Morningstar reported that in the previous year, active U.S. equity funds lost $91.9 billion in outflows while passive U.S. equity funds drew in $156.1 billion. "A clear pattern has emerged this year," wrote senior analyst Alina Tarlea, "consistent outflows on the active side and inflows on the passive side."



One beneficiary has been Vanguard: On Sunday, The Wall Street Journal reported that investors poured $216 billion into the biggest provider of index-tracking products, a record inflow for any mutual-fund firm.

ETFs as a category are also benefiting. The number of ETFs, which trade like stocks but hold a basket of assets like a mutual fund, grew from 113 in 2002 to nearly 1,300 in 2013, according to the Investment Company Institute. And last month, they hit another milestone, surpassing $2 trillion in assets. While that's still dwarfed by the $16 trillion in mutual funds, ETFs are growing at a faster rate. In 2010, they had just half the assets they do now.

Actively managed funds have another problem when compared to index-fund investing and ETFs: in general, their fees are higher. While fees on all funds have come down over the years, average fees on actively managed equity funds were 89 basis points in 2013, compared with 12 basis points for index equity funds, according to the Investment Company Institute. (When it comes to the cost of index funds relative to the cost of ETFs, a study by Alex Bryan and Michael Rawson of Morningstar found that "the difference in expenses between the two vehicles is small.")

Still, some active fund managers out there do beat the market. Morningstar has nominated the managers of several domestic stock funds, including the American Century Mid Cap Value Fund and T. Rowe Price Mid Cap Growth, as contenders for domestic stock fund manager of the year. These funds outpaced the vast majority of their peers and beat their benchmark indices.

"It's hard work separating the wheat from the chaff" in the world of actively managed funds, said Stephen Horan, managing director at the CFA Institute, but winners do exist.



Read More How many 401(k) choices are too many?

John Rekenthaler, vice president of research at Morningstar, examined the total returns of different fund management approaches in 2014, comparing actively managed Vanguard funds, which have relatively low fees, to other low-cost active funds, Vanguard passively managed funds, and actively managed funds with high fees. The highest ranked were Vanguard's actively managed funds, followed by the other low-cost active funds and the passive funds. The high-cost actively managed funds were the lowest ranked.

Story continues