With Sydney in the midst of a vital infrastructure boom, community backlash to development needs to be put to bed, according to the Planning Minister.

At a panel discussion on the city’s future on Friday, Anthony Roberts said he was “done with the term overdevelopment” and annoyed at “councils whingeing along the way” as Sydney undergoes a much-needed renovation.

“We needed to move the dial dramatically and we did,” Mr Roberts told those gathered for Sydney 2050: Today’s planning for tomorrow’s Sydney.

With the city’s population expected to swell to 7.5 million people by 2050, Mr Roberts, who is also Housing Minister, said inaction was not an option and more homes had to be built.

Mr Roberts said Sydney was in catch-up mode because past planning policy had “failed future generations with our love of the quarter-acre block” and unwillingness to control or plan for urban sprawl.

“If you think this government is obsessed with infrastructure that’s why, we can’t live like the 1970s when people built homes on subdivisions and then worried about who was going to build their footpath, school or road,” he said.

He said Sydney’s “bumbling planning history” had shown infrastructure and services must meet or exceed housing and population growth.

“Overdevelopment does not exist, infrastructure failure exists,” he told those gathered for the event hosted by the Committee for Economic Development of Australia.

“Admittedly previous governments allowed poor development but I’m confident we’re rectifying that with the world’s best design and planning principles” he said, adding the missing middle — backed by supporting infrastructure — was key to supporting a growing city.

But concerns about whether infrastructure can keep up with the rapid rate of development are rife across both communities and councils in Sydney.

Earlier this year Mr Roberts had to defer the introduction of a policy aimed at encouraging medium-density housing in dozens of councils, due to backlash and concerns about the impact of increased density.

“One person’s overdevelopment is another person’s home and that’s the issue,” he told Domain. “It’s easy for someone to say I don’t want more development, well you’re fine, you’ve got a home, but think about your children and grandchildren.

“Now as Planning Minister I’m allowed to say this, and coming from local government I’m even more qualified to say this, but I’m a tad annoyed occasionally at some councils whingeing along the way, but I’ll cop it.”

The minister said he was happy to give councils more time to amend their planning strategies, and noted he would ultimately like the planning department “to get out of the way of local councils and communities”.

In her keynote address, City of Sydney Lord Mayor Clover Moore said population pressures were challenging infrastructure.

She later told Domain fears of overdevelopment among communities across Sydney were justified, because people were used to seeing development put in place without the supporting amenity and open space.

“Take Mascot for example, people see rows and rows of dense development that have gone up with no open space or facilities… and that’s what worries people,” she said.

Cr Moore said it was only natural for councils, cash-strapped by rate-pegging and cost-cutting, to be concerned about development, as many did not have the resources to provide the facilities needed to support it.

“If [councils] are not seeing the infrastructure going hand in hand [with new development], I think they have every right to be concerned about it,” she said.

Cr Moore said there needed to be closer collaboration and cooperation between the two levels of government and more respect for the role local government had to play in planning.

Northern Beaches mayor Michael Regan told Mr Roberts his council wanted to deliver more housing, but needed other ministers and departments to provide supporting infrastructure.

Earlier this week a report by the Urban Development Institute of Australia revealed delayed infrastructure construction has curtailed the supply of as many as 81,000 homes in NSW.

Mr Roberts agreed housing was being delayed due to a lack of co-operation between government and utility departments and said he would soon announce a taskforce to sort out such issues and encourage better co-operation.