Warren Buffett. AP Images Billionaire investor Warren Buffett, the chief executive of Berkshire Hathaway, dumped on hedge funds during Fortune's Most Powerful Women Summit.

He said that as hedge fund assets grow in size, the fund managers don't really need to worry as much about performance.

That's because hedge fund managers are typically paid through a compensation structure commonly known as "2 and 20," which stands for a 2% management fee and a 20% performance fee.

That means a hedge fund manager would charge investors 2% of total assets under management and 20% of any profits.

When you have $20 billion in assets, you're getting $400 million just from management fees.

"That 20% becomes less important," Buffett said.

"Attracting money is enormously important," Buffett continued, noting that "different styles become popular from time to time."

Activist investing — where an investor builds a large enough stake in a company to aim for a board seat and push for changes from management — has been a popular strategy lately. Buffett has been a known critic of activism.

Innovators, imitators, and swarming incompetents

On Wall Street, Buffett likes to say that there are "the innovators, the imitators, and the swarming incompetents."

"Investment people run around promoting the flavor of the day," Buffett said, adding later that "Activism is a salable form, therefore, it gets sold and Wall Street sells it."

Investors have gravitated toward the activists lately. Those funds saw significant inflows at the beginning of this year. During the first quarter, investors allocated $3.9 billion of new capital to activist funds, bringing the money invested in activists to $127.5 billion, according to data from Hedge Fund Research.

"A lot of money out there [is] willing to pay high fees for the promise of performance. [You] don't have to particularly deliver. [The] promise lasts long enough to get you and your children rich," he said.