To Warren Buffett, it is the one area where he says his investing track record is "awful," "pretty bad" or "really bad."

Berkshire Hathaway Inc. 's tiny gaggle of retail businesses—from See's Candies to Ben Bridge Jeweler and Nebraska Furniture Mart—gets little attention from investors and analysts, and the companies are profitable.

But the retail sector continues to confound the billionaire investor and his partner Charlie Munger. They have bemoaned their bad luck in retail investing for years, speaking about their retail "failures" at annual meetings and in interviews. More recently, the duo—famously averse to technology bets—have lamented how the Internet is rapidly reshaping shopping habits and affecting Berkshire-owned retailers in ways they didn't expect.

At Berkshire's annual meeting in May, Mr. Munger singled out the retail business as one under threat. "I think the new technology is going to be very disruptive...retailing in particular is facing major threats," he said.

That's a lot of air time for nine businesses that account for 2% of Berkshire's $182 billion in annual revenue and shows how conscious Mr. Buffett is of the rare blemish on his lauded investing record.