Men's apparel retailer Men's Wearhouse said it fired Executive Chairman George Zimmer, who founded the company more than 40 years ago, sending the company's shares down in trading on Wednesday.

The company said it had postponed its annual shareholder meeting to renominate the existing slate of directors without Zimmer.

Following the termination, Zimmer told CNBC, "Over the last 40 years, I have built MW into a multi-billion dollar company with amazing employees and loyal customers who value the products and service they receive at MW. Over the past several months I have expressed my concerns to the Board about the direction the company is currently heading.Instead of fostering the kind of dialogue in the Boardroom that has in part contributed to our success, the Board has inappropriately chosen to silence my concerns through termination as an executive officer."

Men's Warehouse issued its own statement, saying: "The board expects to discuss with Mr. Zimmer the extent, if any, and terms of his ongoing relationship with the company."

Despite the near-term uncertainty that the departure has caused, Stifel reiterated its "buy" rating on the stock.

"We believe that despite Zimmer's planned transition to a smaller role at the company, he had difficulty letting go of the reins and the leadership of the business," analysts said.

What is Men's Wearhouse stock doing now? (Click here for the latest stock price.)

Although ZImmer has been the company's spokesperson for decades, Stifel analysts said "the company has the legal right to his image as well as 500 hours of footage of Zimmer."