New York (CNN Business) The world's most well-known investor says it's hard for him to do better than the broader market. So should you even try?

Warren Buffett told CNBC on Monday that he's had a "tough time" trying to beat the S&P 500. The Oracle of Omaha, who just released his annualshareholder letter, suggested that the index is still the best way to invest in the stock market for most people. He even joked that most of the time he doesn't know how to pick individual stocks.

Buffett told the network that his two investing gurus, Ted Weschler and Todd Combs, have each underperformed the S&P 500 during the past few years by a "tiny bit." Even so, he added that their stock picks have done better than his.

Of course, the long-term track record of the world's third-richest man says otherwise. But he has a point.

Taking less risk

It seems silly for investors to try picking and choosing the best stocks — the market, writ large, historically tends to go higher. Just ride the wave, and you'll do fine.

But why the S&P 500 and not the Dow Jones Industrial Average? The S&P 500 may not be as famous as the Dow. But this blue chip index has exposure to nearly all of America's most important companies, and it ranks them by how large they are.

That makes it a much better proxy for the US economy than the Dow, which has only 30 stocks in it and weighs them by their stock prices.

Unsurprisingly, the S&P 500 now skews toward technology. The sector has done extraordinarily well over the past decade because of growth in online advertising, social networks, e-commerce, cloud computing and mobile gadgets.

Microsoft MSFT Apple AAPL Amazon AMZN Google GOOGL Facebook FB The four most valuable companies in the S&P 500 areandowner Alphabet.ranks sixth. The only non-tech company among this group of giants: Berkshire Hathaway, which is in fifth place.

Berkshire Hathaway is famously underweight the technology sector, however.

JUST WATCHED Here's why Warren Buffett trusts Tim Cook Replay More Videos ... MUST WATCH Here's why Warren Buffett trusts Tim Cook 01:42

IBM IBM Yes, Buffett's top holding is now Apple. But before Berkshire invested in the company, its favorite tech stock was— not exactly the most exciting name in the industry.

Bank of America BAC Wells Fargo WFC American Express AXP Most of Berkshire's other top stocks are far less dynamic than Apple, too. Big financial firmsandrank among its five largest investments.

Coca-Cola KO Kraft Heinz KHC Southwest LUV Delta DAL United Continental UAL American AAL Buffett is also a fan of consumer staplesand struggling . And his firm has recently fallen in love with airlines:andare among Berkshire's top 20 holdings.

All of that means it should not be a major surprise that the Berkshire portfolio has lagged the market lately.

Better than bonds

Instead of looking at Berkshire, investors should focus on what Buffett thinks about the stock market broadly. He noted that the market is cheap, especially with longer-term US Treasury bond yields like the 30-year hovering around 3%.

Buffett told CNBC that faced with a choice between bonds and stocks, the decision is easy.

"I'd buy the S&P in a second," Buffett said, adding that he would give "enormous odds" that the S&P 500 will do better than 30-year bonds over the next three decades.