The Royal Borough of Kensington and Chelsea has approved plans for a half-billion pound luxury retirement complex that includes just five affordable homes at a time when 11 families who survived the Grenfell Tower fire are still living in hotels 18 months on.

The Conservative controlled council granted consent for the scheme on a prime site in the south of the borough that includes 142 homes, some of which will be let for up to £10,000 a month.

Dubbed “caviar care”, the scheme is designed to appeal to multi-millionaire downsizers and includes three town houses expected to sell for about £12m apiece.

The council and the developer argue that it is allowable under planning rules because the properties are classed as “extra care” homes, regardless of how expensive they are. The sale value of the mostly one-bedroom and two-bedroom flats averages £3.6m each. The developer is also marketing another luxury retirement complex nearby featuring a restaurant serving £250 pots of caviar.

The consent comes amid a growing argument over affordable housing in the capital between the Labour mayor, Sadiq Khan, and the Conservatives. Khan said he was “extremely disappointed” at the amount of affordable housing as part of the retirement development, a factor he said was “unacceptable”.

Khan also attacked the housing secretary, James Brokenshire, for threatening to block a planning application for a separate development in the borough that would have 35% affordable homes. Brokenshire countered by accusing Khan of failing to tackle the housing crisis, saying he “simply doesn’t understand how the housing market works”.

There will only be five affordable homes in the complex on the site of Heythrop College, a former Catholic theological school. The developer, Westbourne Capital Partners, argues that the affordable housing contribution of five homes exceeds targets as it amounts to 54% of housing floor space, which its measures against the three homes that are for sale rather than the 142 properties in total. It has agreed to pay £8.7m towards community infrastructure funds and £4m to make the High Street Kensington tube station more accessible.

The housing will be open to anyone over the age of 65 if they can afford the rent. Younger buyers will have to show they have a care need. Three hours a week care is provided as part of the rent.

Johnny Sandelson, director of Westbourne, said the project provided “a social benefit” by meeting the need for high-end retirement homes and freeing up existing housing stock in the area. “We have provided the amount of affordable homes needed,” he said.

The council’s 2016 planning brief for the site stressed the need for “inclusive and mixed communities”. The borough has a housing waiting list of 3,546 households. A quarter of the 203 households made homeless by the Grenfell fire have not yet been permanently rehoused a year and a half after the disaster.

“The need for residential care for older people is decidedly not for very high-income residents,” said Emma Dent Coad, the Labour MP for Kensington. “The overwhelming need is for low- to middle-income older residents. Even our higher-income residents would find the cost of these units exorbitant.”

The council said the scheme met “demand in the borough” for more elderly care accommodation, adding that nearly 300 affordable homes had been built in the borough between 2014 and 2017, and that it was planning to build 600 homes itself.

Khan has accused Kensington and Chelsea of failing to meet targets for new affordable homes.

“Last year no affordable homes were given planning permission by the council,” a spokesman for the mayor said. “In the year before, the council approved just nine affordable homes – just 3% of its target of 293.”

In March, Khan took control of the planning process for a separate 17-storey tower in Notting Hill Gate, London, which had been rejected by the council. He granted consent for the Newcombe House project with 35% social housing, but the Ministry of Housing last week indicated it might block the plans which some locals have opposed because of the building’s height.

“This single development alone will provide almost three times as many new affordable homes as the borough has approved across all other planning applications in the last two years put together,” said Khan. “Londoners who are being left to languish on the borough’s housing list will feel let down that this scheme is now in jeopardy.”

But Brokenshire told the Guardian: “There has been a 20% fall in housing delivery across London in the past year, illustrating the mayor’s lack of will, lack of urgency and lack of determination to get on and build houses for Londoners. In his first 18 months as mayor there were 20% fewer housing starts than under his Conservative predecessor ... He is not spending enough time on the issue and delivery has collapsed under his tenure. Major developers are looking twice at London as a development opportunity.”

London’s deputy mayor for housing, James Murray, described the comments as “a desperate attempt at political distraction by the secretary of state, who is looking to block plans for new homes, including those at social rent levels, which Sadiq had given the go-ahead to”.

Murray added: “It is telling [that Brokenshire] doesn’t even mention affordable housing – probably because he knows Sadiq started more homes at social rent levels, and more genuinely affordable homes overall, last year than in any year since devolution to London.”