“State Street is the critical street in Downtown Madison,” he said. “The architecture is a significant part of it.”

Inspectors have given the building’s owner, Fitchburg-based SCK Investment, two weeks to remove the paint with the help of a chemical paint-removing treatment. The hope is that removal can be done without damaging the soft stone, Cream City brick and mortar. Workers initially tried to power-wash it off, and “proceeded to do some damage” to about a 3-by-3-foot section of the building, city building inspector George Hank said.

To do it right will be to do it “very carefully,” he said. “This is going to take some elbow grease.”

Levitan, also a local historian, said: “Once brick gets painted, it never comes back all the way.”

Madison housing inspection supervisor Kyle Bunnow said the person renting the building wants to open a coffee shop there, and it was that person who had the painting done.

A man doing work inside the building on Wednesday afternoon declined to provide his name but said there had been a “miscommunication” with painters, who thought they were supposed to paint the outside as well as the inside of the building. The inside was also painted mostly black.