The government will not change its immigration policy to try to curb the housing crisis, Prime Minister John Key says.

Reserve Bank deputy governor Grant Spencer has said there has been an inflow of 160,000 people in the past three years, and that cannot be ignored.

He said the government's policy needed to be reviewed to see whether the right people with the right skills were coming into the country.

But John Key said the majority of immigrants were from categories that could not or should not be changed.

"We're going to stick with the plan we've got" - John Key

He said the government did not want to stop returning New Zealanders or Australians buying homes, and it did not want to shut out foreign students who were valuable to the economy.

He said other groups, such as those on temporary visas, were not big home buyers so there was very little change that could make a difference.

Earlier today New Zealand First warned the Reserve Bank might be accused of being "racist and xenophobic" when it said immigration was putting pressure on the housing market.

New Zealand First leader Winston Peters said the Reserve Bank's comments showed people finally realised immigration was a problem.

Photo: RNZ / Alexander Robertson

"When I've said it for a long time, all I've got back from the other political parties, and dare I say it the media, is that 'this party New Zealand First is racist and xenophobic'.

"Now what are you going to say to the governor of the Reserve Bank or the deputy governor of the Reserve Bank when he makes a similar speech?" Mr Peters said.

However, Economic Development Minister Steven Joyce said that was not necessary as the government "fine-tuned" its immigration policy all the time.

Photo: RNZ / Alexander Robertson

"He [Grant Spencer] also points out that the bulk of it is actually Kiwi's returning home and not leaving. And that's been the really big swing.

"He goes on to say there's not much you can do about that and he's right. And he just says we should look at reviewing those skill mixes which we do constantly," Mr Joyce said.

But, Labour Party finance spokesperson Grant Robertson said something had to be done.

"Clearly at the moment it needs to be reviewed. The Reserve Bank has been clear that this is one of the issues that needs to be sorted. It is only one.

Photo: RNZ / Rebekah Parsons-King

"And if it [the Government] only focusses on immigration it won't solve the problem. But it if ignores immigration, it won't solve the problem either," he said.

Last night, the Reserve Bank also signalled tougher lending limits to property investors outside Auckland could be in force by the end of the year.