The U.S. stock market has generally been doing well under President Donald Trump, but the President warned TV audiences on Thursday morning that that would change were he to have to leave office. "If I ever got impeached, I think the market would crash. I think everybody would be very poor," he told Fox & Friends.

Experts are skeptical. "It's a ridiculous remark — the kind of thing a Latin American dictator or a Middle Eastern strongman would say to keep supporters in line," Josh Brown, CEO of Ritholtz Wealth Management, tells CNBC Make It.

Brown, a CNBC contributor, points out, "President Pence will be just as friendly to corporations and just as supportive of low taxes, as would any other Republican." So there's little reason to lose sleep over your investments.

In fact, political events have far less sway over the markets than economic factors, Joseph Davis, Vanguard's global chief economist, tells CNBC Make It. "The history of the stock market shows the more meaningful events for investors and for market influences over a six-month basis or annual basis are economic fundamentals, such as corporate earnings and changes in Fed policy," Davis says.

To be sure, change can cause market disruption because it increases the "ambiguity premium," Kent Smetters, Wharton professor of business economics and public policy, tells CNBC Make It. "But something like an impeachment probability gets priced in early since it is a long, drawn-out process. So whether markets like it or not, it won't likely lead to a sudden change," Smetters says.

Historical trends are mixed when it comes to impeachments. The S&P 500 did fall about 19 percent ahead of of Kenneth Starr's 1998 report on President Bill Clinton's possible perjury and obstruction of justice, but it went on to recover completely and even set a new all-time high in advance of the Senate finally acquitting Clinton in February 1999.

"This time around, while the current crisis may trigger a modest decline, we do not think it will lead to recession and therefore will not result in a new bear market," Sam Stovall, chief investment strategist for investment firm CFRA, tells CNBC Make It.

Even if the U.S. stock market did crash, it would not directly affect "everybody," as Trump contends. In fact, today, investments in the market are heavily concentrated among the richest U.S. families. Prior to the 2007 recession, the top 10 percent of U.S. households owned 81 percent of the stock market. Today, they own 84 percent of it, according to research by New York University economics professor Edward Wolff.

That's not to say a downturn would have no impact on most people, Brown cautions. "When people feel wealthy because of high portfolio values or increased compensation at work, they're more likely to buy more clothing, cars, etc., and go out to restaurants and hire contractors to do work on their houses," he says.

Davis agrees, saying that a significant market correction or crash would have ripple effects that trickled through all levels of American households.

But, he adds, investors should not be losing sleep over the headlines. "If they're stressing over the current market environment, it would be a reminder to evaluate their longer term [investment] plan," he says.

Don't miss: 12 essential money tips for every phase of your life

Like this story? Subscribe to CNBC Make It on YouTube!