Bogle helped bring investing to the masses through his creation of the index fund and founding of the no-nonsense, low-fee investing firm Vanguard Group. He preached a philosophy of steady investing and avoidance of market timing.

"Jack did more for American investors as a whole than any individual I've known," Buffett told CNBC's Becky Quick . "A lot of Wall Street is devoted to charging a lot for nothing. He charged nothing to accomplish a huge amount."

"He converted, in a 30-year period, a lot of people to the right religion of investing," added the Oracle of Omaha. "And it's a good religion. It pays off."

Buffett, 88, has long been an admirer of Bogle because of their shared interest in sensible long-term investing.

The chairman and CEO of Berkshire Hathaway shared these words on Bogle in his widely read annual letter to shareholders two years ago:

"If a statue is ever erected to honor the person who has done the most for American investors, the hands down choice should be Jack Bogle. For decades, Jack has urged investors to invest in ultra-low-cost index funds. In his crusade, he amassed only a tiny percentage of the wealth that has typically flowed to managers who have promised their investors large rewards while delivering them nothing – or, as in our bet, less than nothing – of added value. In his early years, Jack was frequently mocked by the investment-management industry. Today, however, he has the satisfaction of knowing that he helped millions of investors realize far better returns on their savings than they otherwise would have earned. He is a hero to them and to me."

WATCH: Vanguard Founder Jack Bogle dies at 89