Residents of a prime Sydney waterfront location fear their quiet suburb will be turned into a building site as the next stage of a major rail project gets under way at Lavender Bay.

The $9.6 billion line under Sydney Harbour is the next part of construction of the Sydney Metro from Chatswood to Sydenham.

Tunnel boring is scheduled for next year, with Blues Point nominated as the northern acess point to the harbour.

About 8000 tonnes of soil will be excavated at the site, adjacent to Harry Seidler's Blues Point Tower, to create a shaft for the tunnel boring machines.