Construction workers scrambled up the scaffolding of a CBD worksite to escape a "wall of flame" spreading up the face of an office building on Thursday morning, witnesses say.

Another witness said there was a "huge amount of damage on the facade of the building" after the fire broke out near Legal Grounds Cafe, not far from Parliament House on Macquarie Street, about 7.45am.

The fire at a construction site on Macquarie Street. Credit:Twitter/@tsdheo - Douglas Thompson

Macquarie Street was closed in both directions while 12 fire crews attended.

Barrister Peter King, who works in the building next to where the fire began, said he saw the fire burn its way up the plastic covering the outside of the scaffolding.