Two street gangs have clashed violently in Melbourne's CBD, sending terrified innocent bystanders running from Federation Square and a Swanston Street cafe.

Others were forced back inside a restaurant for their own safety as a lengthy brawl erupted in Federation Square, where hundreds of families had gathered to celebrate the Moomba festival and watch the annual fireworks, about 10pm on Saturday.

"We were absolutely terrified," said Troy West, from Bendigo, who spent 45 minutes locked inside a Federation Square restaurant as police tried to contain the rival groups outside.

"You're there for a night out and you don't expect to see gang violence."