A panel has fallen from the top floor of a high-rise building in Sydney's CBD and landed with an "almighty bang", raining shards of glass onto a busy intersection at lunch time, witnesses say.

The incident occurred at the corner of Market and Sussex streets in the city just after midday.

A panel fell from the Allianz Centre in Sydney's CBD, witnesses say. Credit:Nicholas Rugg

Nicholas Rugg said he was standing at the lights waiting to cross the road when the panel fell from the top of the Allianz building and crashed onto the roof of the Fitness First gym below.

He said the panel hit the gym with an "almighty bang" and glass sprayed over the road and onto the roofs of cars waiting at the lights. Another witness claimed that small shards of glass hit a number of people.