Warren Buffett has been at the helm of Berkshire Hathaway (NYSE:BRK-A) (NYSE:BRK-B) for decades, and his track record there makes him one of the most successful investors in history. His long-term performance is nearly unmatched, and those who've stuck by the Oracle of Omaha over the years have gotten big rewards for their loyalty.

Yet recently, Berkshire has underperformed the broader market by a significant margin. In 2019 alone, Berkshire stock is up just 1%, compared to a nearly 17% rise in the S&P 500 index. Although there are undoubtedly many different factors at work to explain Berkshire's underperformance, there's also a key factor that many investors don't even realize is happening that could have a dampening effect on the stock's upward price moves.

Berkshire, Buffett, Bill Gates, and big gifts (and sales)

It's been 13 years since Warren Buffett announced that he would give away the lion's share of his fortune gradually over the course of the rest of his lifetime. The Berkshire CEO started out with an initial gift of Berkshire stock worth $1.5 billion, and from there, annual gifts came at regular intervals. For instance, earlier this year, Buffett gave away roughly 16.8 million Class B shares of Berkshire, worth roughly $3.6 billion at the time. The bulk of those shares -- about 12.8 million -- went to the Bill and Melinda Gates Foundation.

Meanwhile, for the Gates Foundation, having huge holdings of Berkshire stock has been an excellent source of funds. Rather than selling off in one fell swoop the large blocks of stock that it receives as donations, the Gates Foundation has instead made regular sales of shares. For instance, in the first quarter, Gates Foundation holdings of Berkshire B shares dropped by 5 million. Another 5 million B shares got sold during the second quarter.

Downward pressure on the stock?

The pace of the Gates Foundation's sales isn't so large that it would have an immediate and obvious impact. Average daily volume for Berkshire amounts to roughly 4 million Class B shares. If you assume equally spaced sales happening constantly during the quarter, 5 million Gates Foundation-held Berkshire shares would equate roughly 80,000 to 85,000 shares per trading session. That's just 2% or so of typical volume.

Neither does it appear that Berkshire's particular niche has done poorly this year. Financial stocks broadly are up almost as much as the S&P 500, and many prominent insurance companies have actually beaten the market, let alone Berkshire's returns.

The likely answer

Yet longtime investors in Berkshire have seen this story play out before. From time to time, Buffett's philosophy simply doesn't lead to favorable short-term results. Shareholders have to endure a period of underperformance -- sometimes extreme -- that calls into question Berkshire's entire way of investing. Eventually, though, fundamentals reassert themselves and Berkshire's stock catches up to its peers.

For me, that first round was in the late 1990s. Buffett refused to invest in go-go internet stocks, and his returns suffered. When the tech bust of 2000 to 2002 hit, Berkshire's businesses were rock-solid, and the stock clawed back the ground it had lost during the 1990s bull market.

The same thing happened to a more limited extent on several other occasions. From 2003 to 2005, Berkshire dramatically underperformed, but better performance in 2006 through 2008 helped to offset its earlier lagging returns. Berkshire has also seen isolated poor years, such as 2015, but they've often gotten followed by big gains like the stock enjoyed in 2016.

Don't miss the big picture

It's always frustrating when a stock you own doesn't do as well as the market -- especially a stock that's as well-known as Berkshire Hathaway. But if you give up hope just because of a year or two in which a stock trails broader stock indexes, you'll often miss out on the gains that result when Berkshire's mode of investing comes back into style.