Ormond Gigli, longtime photographer for TIME, LIFE, Paris Match and others, snapped everything from farmers to movie stars in his decades long career. But his most famous image, “Girls in the Windows” — taken on New York’s East 58th Street in 1960 – was made on a whim.

Widely considered one of the most famous fashion shots of the 1960s, it captures a slice of long-gone New York (the brownstones pictured were demolished the next day), and the picture’s influence stretches beyond photography. TIME caught up with Gigli before the mid-November release of a new book looking back at his life’s work.

Ormond Gigli

Girls in the Windows wasn’t done on assignment, how did it come about?

No, it wasn’t an assignment. I had a brownstone [studio] that was right across from it on East 58th Street, and I look out the window one day and I see that they are tearing down the brownstones opposite me – they were old and no one was in them. And I’m looking at them and I’m saying: “It’s a shame, you know, what can I do with it?”

I had a great staff there [at my studio], so I’m discussing it with my studio manager – if we could get the frames out of the windows, we could shoot a girl in each window. So I had my studio manager go to talk the head demolition guy, and he said “yes, but you have to put my wife in the shot!”

We had to do it the next day, at 12 o clock when the workers had their lunch hour. We got models, we got friends. They wore their own outfits, nothing was styled – it was a happening.

Nice. So you shot from a window across the street?

I’m actually on the fire escape on the second floor – we had large fire escapes, almost like a balcony or something – I’m set up there with my camera and I’m directing. I had a bullhorn, and I got worried after a while so I said “don’t step out onto the ledges whatever you do!” Because with brownstones, the ledges, without anybody on them can fall down.

What camera did you use? And how many shots did it take to produce the image we know today?

It was shot on a 4×5 Speed Graphic – which I seldom used – with a wide angle lens. And I did about, I’m guessing here, 15 or 18 shots.

I believe not only was the demolition guy’s wife in the shot, yours was too?

Yes! My wife is on the right hand side. If you go up to the second floor she’s two over in a pink outfit.

Girls in the Windows: And Other Stories by Ormond Gigli is available now from powerHouse Books