A Sydney council has lost a $100,000 legal battle against a massage parlour allegedly operating as a brothel after a court found it had failed to prove there were enough prostitutes working there.

The Hornsby Massage Centre is located next door to a primary school tutoring centre and just 50m from Hornsby Girls' High School, Fairfax media reports.

Hornsby Council paid a private investigator to engage the sexual services of a prostitute there as part of its year-long court battle but a North Sydney Local Court judge ruled that the evidence fell short of proving it was a brothel, which by definition requires more than one prostitute to be working onsite.

The landmark ruling means that NSW local councils will be required to pay for multiple trips to suspect businesses in order to get the evidence necessary to close them down.

Hornsby Council is now planning to lobby the NSW Department of Planning and Environment to "urgently review" the legal definition of a brothel.

Cr Nick Berman confirmed to Fairfax the council spent more than $100,000 "trying to do the right thing - and lost".

"To have to invest ratepayer's money to pay private investigators to have sex with prostitutes is, in itself, ludicrous. But to now have to send two, three, even four men in is bordering on the unbelievable."

The landmark decision has prompted Premier Mike Baird to request a parliamentary inquiry into the regulation of NSW brothels.

"We want to make sure the whole of the community is protected and it is clear that something must be done," Premier Baird said.