As if the arrest of 18 key staff in China was not enough, James Packer and his gaming company, Crown Resorts, face a new challenge next week to his plans to develop a high-roller casino on the waterfront at Barangaroo in Sydney.

On Tuesday, the Land and Environment Court will begin hearing a case brought by the Millers Point Fund Inc, a group representing the residents of nearby Millers Point. The area was once home to people living in social housing but nowadays it has got its fair share of well-heeled unit dwellers, some of whom face living, literally, in the shadow of the 271-metre tower Crown plans to build.

If the residents are successful, it could set back the timetable for the casino by months, if not years.

In what promises to be a David and Goliath struggle, the NSW Environmental Defenders Office has taken on the residents’ case against the casino mogul, developer Lendlease, the NSW government, and its decision-maker, the Planning Assessment Commission, arguing that the commission made a fundamental error in its reasoning, which is so grave as to make the planning approval invalid.

“What the Planning Assessment Commission did was say: ‘Sorry, parliament has already decided where this casino should go’,” the chief executive at the Environmental Defenders Office, Sue Higginson, said. “But nowhere in the planning legislation does it say it’s subject to the Casino Control Act.”

Residents are hoping this will prompt a last-minute rethink of the area known as Barangaroo South.

“In our submission we argued that Packer’s tower should be moved close to Hickson Road and the area where he wants to build it should revert to foreshore park,” said Millers Point Fund spokesman John McInerney, a former City of Sydney councillor.

“We think it should be tucked back into the cityscape, instead of on the edge of the harbour, which has traditionally been low rise.”

If the residents are successful, it could set back the timetable for the casino by months, if not years, and potentially force a relocation.

Coming on top of Crown’s problems with the Chinese government and its future ability to bring high rollers to Australia on organised gambling programs known as junkets, that may just prove a bridge too far for Crown.

But let’s backtrack first a few months, then a few years, to understand the story so far.

In April this year a three-person Planning Assessment Commission, delegated with the power to approve major developments in NSW, held public hearings.

Lendlease had applied to alter the master plan for Barangaroo, so the planned hotel on a pier protruding into Darling Harbour could be moved back onto land. Crown was also seeking approval for the 271-metre Crown Tower, which would include a hotel, luxury apartments and a casino.

The hotel over the water had been contentious and the Coalition had campaigned in the 2011 state election for a review. So it was no surprise that when Barry O’Farrell won, he asked Lendlease to ditch it.

But that led to another chain of events. In 2012 James Packer paid Lendlease $2 million for the exclusive right to develop a new hotel on land. Crown then applied to the NSW government with an “unsolicited bid” to build a six-star hotel and casino, initially on land reserved as a public park, and later on another prime waterfront site in the Lendlease area.

The promised $2 billion project would give Sydney one of the “great hotels in the world” and an “iconic” building, Packer said. And because of his exclusive dealing arrangement, he was the only one who could build it.

To make it work as a six-star hotel, the Packer camp argued they needed a casino licence.

“The VIP gaming facilities are an integral part of the Crown proposal” and “necessary to make this project commercially viable”, Crown said.

The chips were already falling in Packer’s favour. In January 2012, before the billionaire lodged his bid, the government revised its guidelines for unsolicited bids, removing the requirement for “independent evaluations” of a project seeking to bypass a tender.

There was also a question mark, which remains unanswered, about what made a casino project so innovative, other than Crown’s two-year exclusive dealing option.

The O’Farrell–Baird government, which won office with the slogan “Open for Business”, simply accepted that Sydney desperately needed more hotel rooms and the only way to get them was to go with Packer.

At the last minute, rival casino operator, Echo Entertainment, operator of The Star, lodged its own unsolicited bid but failed. The upshot was James Packer was awarded a restricted gaming licence for a VIP casino on publicly owned land without tender.

The Casino Control (Amendment) bill, introduced in late 2013, outlined the broad terms of the restricted licence: a ban on poker machines, minimum bets,

a membership regimen to limit entry, and an exemption from the ban on smoking indoors. Also attached to the bill was a small map that showed the geographic area of the casino licence, as required by the legislation. It was located right on the waterfront as Crown desired.

After a heated late-night debate, the bill passed with the backing of everyone except the Greens and the Christian Democrats.

“At no point did any MP raise any issues of planning,” said the Greens planning spokesman, David Shoebridge, who took part in the debate. “The government MPs certainly didn’t talk about planning; the discussion was about licensing.

“It was an ugly exercise in the realpolitik of NSW. Both major parties had been duchessed by Crown so there was no real discussion about whether Sydney needed another casino.”

Meanwhile, Packer worked with architects Wilkinson Eyre to develop a casino tower worthy of the badge “iconic”.

Crown and Lendlease proposed it would be located right on the waterfront, with just 30 metres reserved for a public domain.

When the scale of the Crown tower was revealed, atop a podium housing the casino floors, it prompted a furious outcry from Sydney’s architectural community.

City of Sydney Council branded it a travesty of planning, which would destroy the original concept of a continuous waterfront park open to the people of Sydney. A government-appointed design panel also had qualms, warning that the six-storey podium housing the casino would dominate the public space.

Wisely, the government had decided it would delegate its decision-making power to the independent Planning Assessment Commission.

The objections reached a crescendo at the commission’s public hearings. Three former government architects pleaded with the commission to move the building back from the waterfront and preserve the foreshore park.

It seems they wasted their breath.

When the Planning Assessment Commission issued its report, it said it had “a great deal of sympathy” for the views that the public domain was being compromised.

But it said its hands were tied.

“It is important to understand that the NSW Parliament effectively settled the issue about the location of Crown Sydney when it legislated for the restricted gaming facility on the foreshore on Block Y in 2013,” the members said. “The commission has no power to direct relocation of the restricted gaming facility or to change the associated legislation. It accepts that the licence will remain in the location currently mapped.”

Which brings us to next week.

Shoebridge says the map was barely mentioned during the bill’s debate. It was certainly not made clear that the casino control legislation would effectively gazump the planning process.

And why did the government spend millions of dollars on a design excellence review and public hearings if the most important element – the siting of the casino on the waterfront – had already been set in stone?

It’s now for the Land and Environment Court to sort out whether the approval is valid.

“Lendlease has worked with Crown, the Barangaroo Delivery Authority and other stakeholders to achieve an appropriate planning outcome for the Barangaroo South precinct,” a spokeswoman said. “Lendlease has received approval of a modification to the concept plan applying to Barangaroo South from the Planning Assessment Commission. Lendlease intends to defend that approval in the Land and Environment Court.”

A spokesman for Crown said the company would also be vigorously defending the case.

In the meantime, the stakes have got higher for Packer and Crown. It’s already suffering the effects of the crackdown in China on corruption and moving money out of the country. Last month Crown revealed VIP gaming in Australia was down 8 per cent, to $879 million, and the profits from its one-third share in Melco Crown in Macau down 65 per cent.

Crown has sought to reassure the markets that VIP gaming by high rollers makes up just over a quarter of its revenues, and that China makes up only half of its overseas visitors.

But that’s now. The business model for Crown Sydney, due to open in 2021, is for a restricted casino with bets to start at $30 a head – that is, a casino almost entirely focused on visiting high rollers.

That adds at least two questions to the ones the Land and Environment Court is considering on Tuesday. Will Packer seek to have the licence conditions changed? Will he rethink the entire project?