Mr. Rosen is straightforward in court papers: The building is his and he can do what he likes.

In 1989, the Landmarks Preservation Commission designated both the Seagram Building and the Four Seasons restaurant as city landmarks, memorializing the contributions of Mies van der Rohe, the architect who designed the building, and Philip Johnson, who designed the restaurant.

But the commission excluded the curtain, which depicts a bullfight and is known as “Le Tricorne,” because it was owned separately and could be moved at any time.

“They’ve elevated this into something that it shouldn’t be,” Mr. Rosen said of his former allies at the conservancy. “It’s about a landlord’s rights. I want to do the right thing. I need to remove it and restore it. Then I will make a decision on what happens to it in the future.”

Mr. Rosen, who has an art collection of more than 800 works worth, he says, “well over $500 million,” also resents being painted as a Picasso-hater and scoffs at those who say he referred to the tapestry as a “schmatte,” Yiddish for rag.

“Everybody says I hate Picasso,” said Mr. Rosen, who was appointed chairman of the New York State Council on the Arts by Gov. Andrew M. Cuomo. “But I live with five of them in my home, two in my bedroom.”