The next phase of Ralph Lauren's reinvention won't involve its Fifth Avenue Polo store.

The luxury brand on Tuesday said it will shutter the high-profile location, just a stone's throw from Trump Tower. Ralph Lauren will meanwhile restructure its digital operations and trim its workforce in a plan that will save $140 million annually.

Those anticipated savings are in addition to the annual $180 million to $220 million it announced in June. At that time, the company said it would trim roughly 8 percent of its full-time headcount, close more than 50 stores and focus on its three core brands. That includes the Polo label.

Its latest actions will result in a $370 million charge, the company said.

"These are important actions we are taking to continue our evolution," CFO Jane Nielsen said in a statement.

A spokeswoman for Ralph Lauren did not immediately respond to CNBC's questions regarding whether the decision to close its Fifth Avenue flagship was related to a disruption in business because of its proximity to Trump Tower. The building has been under heavy security and has been the site of protests both during the presidential campaign and following Donald Trump's election.

Tiffany said last month that sales in its New York flagship store declined 11 percent for the full year, including a 7 percent drop in the fourth quarter. That store is adjacent to Trump Tower.

"We continue to review our store footprint in each market to ensure we have the right distribution and customer experience in place," Ralph Lauren's Nielsen said in a statement. "The decision will optimize our store portfolio in the New York area and allow us to focus on opportunities to pilot new and innovative customer experiences."

After closing its Fifth Avenue Polo shop, the company said it will still operate seven stores and its Polo Bar Restaurant in New York City. It will integrate the product from its shuttered store into these other locations.

The company will also test new retail concepts, including the potential to grow Ralph's Coffee. It opened the first of those locations in its Polo flagship in 2014.

The latest changes at Ralph Lauren come just two months after the retailer said CEO Stefan Larsson would exit this coming May after less than two years in the role. Nielsen will drive the company's strategy until a new CEO is named.