"It's unfortunate. These people came to Washington trying to help the government. What's going on now, they can't. So they have to walk away because their No. 1 priority is to protect the brand they represent," he said.

In an interview with " Closing Bell ," Weber said it was time for them to walk away.

Top business executives disbanded the Strategy & Policy Forum on Wednesday in response to Trump's reaction to the deadly white supremacist rally in Charlottesville, Virginia. Several CEOs also quit the president's manufacturing council before Trump dissolved it.

The CEOs who abandoned President Donald Trump's business councils had to think of their companies first, former LVMH CEO Mark Weber told CNBC on Thursday.

President Donald Trump speaks during a meeting with manufacturing executives in the Roosevelt Room of the White House in Washington, Thursday, Feb. 23, 2017.

Trump's remarks about the rally over the weekend have been widely criticized. On Tuesday, he blamed "both sides" for the violence that left one person dead and several others injured.

He also said some of the people participating in the rally, which featured swastikas and chants of "Jews will not replace us," were only against the removal of a statue of Confederate Gen. Robert E. Lee. Then on Thursday, he tweeted against removing the "beautiful" Confederate statues and monuments.

Ben Baldanza, former CEO of Spirit Airlines, agrees that executives need to do what is right for their companies.

"Two things that businesses don't like are uncertainty and distraction and Washington is sure supplying plenty of both of those things right now," he told "Closing Bell."

He hopes Trump and his advisors stay close to the business community because if the president is unable to get the support he needs for his pro-growth agenda, "that's going to make it a very difficult time for American business and the economy."

In fact, just because CEOs are publicly distancing themselves from the president doesn't mean Trump still can't call them for advice, Weber noted.

"Publicly, these folks may not want to put their name out here. Privately, how can you refuse the president of the United States and turn your back on the country? It just won't happen," he said.

— CNBC's Jacob Pramuk contributed to this report.