Charlie Munger, Warren Buffett's right-hand man at Berkshire Hathaway, said that places like California and Connecticut have been very "stupid" for driving rich people away from their states.

"It's been serious. Driving the rich people out is pretty dumb if you're a state or a city," Munger told CNBC's Becky Quick in an interview Thursday. "There are a number of places that have shot themselves in the foot; Connecticut, California, New York City."

Munger was answering a broader question, in the wake of Amazon ditching its New York City headquarters plans, about whether some cities and states need to make their tax structures and regulations more attractive to wealthy individuals and businesses.

In Connecticut, "they've driven out all the rich people. California is doing the same thing. I know a lot of rich people who have left California," Munger added. "I think it's really stupid for a state to drive the rich people out. "They are old, they keep your hospitals busy, they don't burden your schools, police departments or prisons. Who wouldn't want rich people?"

California and Connecticut have two of the highest tax burdens in the country, according to WalletHub. The burden in California is 9.57 percent while Connecticut's is 10.19 percent. To be sure, California is home to some of the largest companies in the world, including Apple and Facebook. Meanwhile, Connecticut is the home of some of the largest hedge funds in the world including Bridgewater Associates.

Recently, some lawmakers have been pushing for higher taxes on the wealthy, especially Rep. Alexandria Ocasio-Cortez, D-N.Y. Ocasio-Cortez has proposed a 70 percent marginal tax on incomes over $10 million in an effort to bridge the growing wealth gap between the rich and the poor.

But Munger thinks the divide will slowly bridge itself as interest rates are unlikely to go "much lower" from current levels. By slashing rates and implementing quantitative easing measures a decade ago, the Federal Reserve inadvertently bailed out the rich to help the poor during the financial crisis by boosting asset prices, Munger said.

"Nobody was doing that because they love the rich; they just didn't have any other tools in the kit," he said. The inequality that came from that "wasn't malevolent and it was an accident and it probably won't happen again."

Subscribe to CNBC on YouTube.

WATCH: Investing legend Charlie Munger on investing, the buyback debate and much more