A four-storey painting of a penis that piqued the curiosity of New Yorkers when it appeared on Christmas Eve was being painted over on Wednesday – by order of the building’s landlord.

The painting, on an apartment building on Broome Street in the Lower East Side, was commissioned by a local street art foundation and made by a Swedish artist, Carolina Falkholt, as a companion to a similarly vast if more abstract vagina, further east on Pike Street.

Falkholt told the Guardian her work was often about “not feeling ashamed of your body and who you are as a sexual being”.

“I usually paint giant vaginas, pussies and cunts,” she said, “and since I had just finished one on the side of a five-storey building, I felt like a dick was needed. The wall space on Broome was a perfect fit for it. To paraphrase [the artist] Judith Bernstein, if a dick can go into a woman, it can go up on a wall.”

On Broome Street on Wednesday afternoon, opinions were mixed. A laundry owner in the mainly Chinese neighbourhood, who declined to offer his name, said the image had made residents uncomfortable.

“We don’t like it, and we hope they take it down,” he said.

Avery Plewis, who was visiting from Toronto, wondered if the painting constituted a kind of visual sexual harassment. “It’s weird, because it’s so phallic that it’s beyond sexual harassment,” she said. “Harassment is often secretive but this is totally in-your-face.”

For others, the phallus provided amusement on a bitterly cold day. Some noted the neighbourhood’s long association with street art and graffiti, including work by Jean-Michel Basquiat, Keith Haring and Richard Hambleton.

“The colors make it very artistic,” said Lauren Deutsch, from Los Angeles. “I’d laugh about it but I wouldn’t want it myself. We don’t see anything like that in LA.”

Three doors down, at Spring Studio, George Petridis was leaving a drawing class where he had been studying Leonardo Da Vinci and anatomy.

“Walking out and seeing this is a massive contrast,” he said. “That’s what some people think of as art right now. It doesn’t offend me, but it doesn’t bother me either, and I’m not going to spend a lot of time thinking about it.”

It turned out he would not need to. An NYC building inspector who had been called out to photograph the image and who did not wish to give their name said “after my superiors look at this, it might be out of our hands. It’s a decision for higher-ups.”

But Katie Grinero, a building manager elsewhere in the city, indicated the likely fate of the painting when she said she considered the image to constitute property damage.

“Imagine if somebody put that on your house without permission,” she said. “I wouldn’t like that. You can [have] it in your own studio, but I don’t think it’s appropriate for the streets with all the kids. It doesn’t teach anything.”

According to the Low-Down NY, the blog that first reported on the installation, community leaders called for the painting to be removed as soon as it was finished.

Naomi Pena, the district 1 community education council president, reportedly wrote to Baby Brasa, a Peruvian restaurant that runs street art foundation The New Allen, which commissioned the work, calling for the painting to be removed. Contractors were soon called in.

In her exchange with the Guardian, Falkholt was philosophical.

“Art,” she said, “is one of the only places left where we can truly be free and discuss whatever difficult topics there are, since art has the ability to translate and transform language in any direction possible.”