A Sydney council has lost a landmark legal battle against a massage parlour it alleged was operating as an illegal brothel, after a court ruled the council failed to prove there was enough sex being provided on the premises.

Hornsby Council paid a private investigator to go undercover inside the parlour and have sex with a prostitute as part of a bitter, year-long legal battle to have the operation closed. The business operates directly next door to a tutorial centre for primary school children - and 50 metres away from Hornsby Girls' High School.

Case dismissed: Pupils from Hornsby Girls' High School walk past a massage parlour that won a legal battle with the council. Credit:Jessica Hromas

But in a benchmark decision, a judge has dismissed the case, ruling that council's evidence of sex being sold on the premises fell short of the NSW's specific definition of the term "brothel" - which requires more than one prostitute to be providing services onsite. The outcome means both Hornsby - and other councils - would have to fund multiple trips inside suspect premises to have any chance of a result.

In response, Premier Mike Baird announced he would ask the NSW Parliament to establish a "full parliamentary inquiry" into the regulation of brothels across the state, saying it was a complex issue and he wanted to get it right.