The Tory party chairman, Grant Shapps, has described as an "absolute disgrace" a call from the United Nations for the government to scrap the so-called bedroom tax.

Accusing Raquel Rolnik, the UN special rapporteur on housing, of having an agenda, Shapps said he had written to the UN secretary general, Ban Ki-moon, demanding an apology and an explanation of Rolnik's findings.

The UN investigator had not been invited to Britain by ministers, and was biased, Shapps said.

"It is completely wrong and an abuse of the process for somebody to come over, to fail to meet with government ministers, to fail to meet with the department responsible, to produce a press release two weeks after coming, even though the report is not due out until next spring, and even to fail to refer to the policy properly throughout the report.

"That is why I am writing to the secretary general today to ask for an apology and an investigation as to how this came about," Shapps told BBC Radio 4's Today programme.

However, the Department for Communities and Local Government confirmed that Rolnik had met Eric Pickles, the secretary of state. She also met council officials in Glasgow.

Rolnik later countered Shapps's claims at a press conference in London, signalling further blistering criticisms of housing and welfare policy in her final report next year.

She had been invited by the UK government, and had met "numerous" government officials in the four countries as well as Pickles and Don Foster, the communities minister, she said.

Rolnik said her recommendations came in a preliminary draft before more detailed discussion with the UK and Welsh, Scottish and Northern Irish governments. "My expectation is that the engagement can be very fruitful, and lead to a better report with full and solid recommendations."

She warned that housing benefit caps would make moving to the private rented sector increasingly difficult for those on low incomes, and complained that homes were now allowed to stand empty in London and elsewhere because they had been sold abroad as financial assets.

The system for helping the poor in Britain had been weakened by "a series of measures over the years, notably by having privileged homeownership over other forms of tenure", said Rolnik.

She cited the government's "help to buy" scheme and failure to replace homes removed from social housing by two decades of tenants' right to buy their council homes, adding that "housing needs not housing markets" should be at the heart of government decision-making.

"It is possible to stimulate the economy and construction industry if you provide more social housing and affordable housing," Rolnik said, adding that such a recommendation would be made in her final report.

"The right to housing is not about a roof anywhere, at any cost, without any social ties. It is not about reshuffling people according to a snapshot of the number of bedrooms at a given night."

Stopping short of demanding the suspension of benefit caps, Rolnik said the government must "put in place a system of regulation for the private rent sector, including clear criteria about affordability, access to information and security of tenure".

She also warned over increasing stigma being shown toward Gypsies, Travellers and Roma struggling to find accommodation. She had concerns too about provision for refugees and asylum seekers.

Rolnik did say Britain had set an example in the way it had renovated old social housing estates and praised its mixed communities and lack of segregation.

Rolnik recommended the abolition of the bedroom tax – which the government calls the spare room subsidy – after hearing "shocking" accounts of how the policy was affecting vulnerable citizens during a visit to the UK.

Britain's record on housing was also worsening from a human rights perspective, Rolnik said in a Guardian interview after presenting her preliminary findings to the government.

The former urban planning minister in Brazil said Britain's previously good record on housing was being eroded by a failure to provide sufficient quantities of affordable social housing, and by the impact of reforms to the benefits system.

After speaking to dozens of council house tenants in Britain during her visit over the past fortnight, Rolnik said she was particularly concerned by the impact of bedroom tax. The policy, introduced by the government in April, charges tenants extra for under-occupying homes that are supposedly too large for them.

Rolnik said she was disturbed by the extent of unhappiness caused by the bedroom tax, and had been struck by how heavily this policy was affecting "the most vulnerable, the most fragile, the people who are on the fringes of coping with everyday life".

During her visit, she travelled to Belfast, Manchester, Glasgow, Edinburgh and London, visiting council estates, food banks, homelessness crisis centres, Traveller sites and new housing association developments. "My immediate recommendation is that the bedroom tax is abolished," she said.

Rolnik has spent much of her five-year tenure as the UN's unpaid special rapporteur on adequate housing looking at human rights violations in countries including Rwanda and Kazakhstan. Appointed by the UN human rights council, the former minister with the centre-left Workers' party spent her previous mission this year looking at slum housing in Indonesia. But Britain's housing crisis was an equally urgent subject for investigation, she said.

"I was very shocked to hear how people really feel abused in their human rights by this decision and [ask] why, being so vulnerable, they should pay for the cost of the economic downturn, which was brought about by the financial crisis. People in testimonies were crying, saying: 'I have nowhere to go,' 'I will commit suicide.'"

During interviews with council officials, she noted that they were struggling to cope with the fallout from the policy's introduction, not least because there was a shortage of single-bedroom properties into which tenants might downsize.

"It's so clear that the government didn't really assess the impact on lives when it took this decision … The mechanism that they have in place to mitigate it – the discretionary payment that they provide the councils with, it doesn't solve anything: it's for just a couple of months, and the councils cannot count on that on a permanent basis. They don't know if it's going to be available next year, so it's useless," she said.

Historically, "the United Kingdom has much to be proud of in the provision of affordable housing", she said, but its reputation was "being eroded from different sides".

The state had an obligation to "put in place safeguards to protect the most vulnerable. And what I am seeing here is quite the opposite: the most vulnerable are having to pay for these cuts." The country was "going backwards in the protection and promotion of the human right to housing".

Rolnik will present a report with her conclusions to the UN human rights council in Geneva early next year.

The bedroom tax could constitute a violation of the human right to adequate housing in several ways, she said: for example, if the extra payments forced tenants to cut down on their spending on food or heating for their home. She said her conclusions should carry weight in British courts, where a number of legal challenges to the bedroom tax were under way.

"It depends on how much the judiciary here takes into account the international legislation. In principle they should, because the UK has signed and ratified the international covenant on economic, social and cultural rights," she said, referring to the document that defines adequate housing as a human right.

A Department for Work and Pensions spokesman said: "It is surprising to see these conclusions being drawn from anecdotal evidence and conversations after a handful of meetings, instead of actual hard research and data. Britain has a very strong housing safety net, and even after our necessary reforms we continue to pay over 80% of most claimants' rent if they are affected by the ending of the spare room subsidy."