The United Nations' special investigator on housing has told the British government it should scrap the bedroom tax, after hearing "shocking" accounts of how the policy was affecting vulnerable citizens during a visit to the UK.

Britain's record on housing was also worsening from a human rights perspective, Raquel Rolnik, the UN special rapporteur on housing, said in a Guardian interview after presenting her preliminary findings to the government.

Rolnik, a former urban planning minister in Brazil, said Britain's previously good record on housing was being eroded by a failure to provide sufficient quantities of affordable social housing, and more recently by the impact of welfare reform.

After speaking to dozens of council house tenants in Britain during her visit over the past fortnight, Rolnik said she was particularly concerned by the impact of bedroom tax, officially known as the new spare room subsidy. The policy was introduced by the government in April, and is designed to charge tenants extra for under-occupying homes that are supposedly too large for them.

Rolnik said she was disturbed by the extent of unhappiness caused by the bedroom tax and struck by how heavily this policy was affecting "the most vulnerable, the most fragile, the people who are on the fringes of coping with everyday life".

During her visit she travelled to Belfast, Manchester, Glasgow, Edinburgh and London, visiting council estates, food banks, homelessness crisis centres, Traveller sites and new housing association developments. "My immediate recommendation is that the bedroom tax is abolished," she said.

Rolnik has spent much of her five-year tenure as the UN's unpaid special rapporteur on adequate housing looking at human rights violations in countries including Rwanda and Kazakhstan. Appointed by the UN human rights council, the former minister with the centre-left Workers' party spent her previous mission this year looking at slum housing in Indonesia. But Britain's housing crisis was an equally urgent subject for investigation, she said.

"I was very shocked to hear how people really feel abused in their human rights by this decision and why – being so vulnerable – they should pay for the cost of the economic downturn, which was brought about by the financial crisis. People in testimonies were crying, saying 'I have nowhere to go', 'I will commit suicide'."

During interviews with council officials, she noted that they were struggling to cope with the fallout from the policy's introduction, not least because there was a shortage of single-bedroom properties into which tenants might downsize."It's so clear that the government didn't really assess the impact on lives when it took this decision … The mechanism that they have in place to mitigate it – the discretionary payment that they provide the councils with, it doesn't solve anything, it's for just a couple of months, and the councils cannot count on that on a permanent basis, they don't know if it's going to be available next year, so it's useless," she said.

Historically, "the United Kingdom has much to be proud of in the provision of affordable housing," she said, but its reputation was "being eroded from different sides". The state had an obligation to "put in place safeguards to protect the most vulnerable and what I am seeing here is quite the opposite – the most vulnerable are having to pay for these cuts". The country was "going backwards in the protection and promotion of the human right to housing".

Rolnik was due to meet Eric Pickles, secretary of state for communities and local government to discuss her investigation. She will present a report with her conclusions to the UN Human Rights Council in Geneva early next year.

The bedroom tax could constitute a violation of the human right to adequate housing in several ways, she said, if for example the extra payments forced tenants to cut down on their spending on food or heating their home. She said her conclusions should carry weight in British courts, where a number of legal challenges to the bedroom tax are under way. "It depends on how much the judiciary here takes into account the international legislation. In principle they should because the UK has signed and ratified the International Covenant on Economic, Social and Cultural Rights," she said, referring to the document which defines adequate housing as a human right.

A Department for Work and Pensions spokesman said: "It is surprising to see these conclusions being drawn from anecdotal evidence and conversations after a handful of meetings – instead of actual hard research and data. Britain has a very strong housing safety net and even after our necessary reforms we continue to pay over 80% of most claimants' rent if they are affected by the ending of the spare room subsidy."