For sport this week I have tried to chase up the deposits of a couple of tenants I met on Twitter. I think some letting agents see it as a matter of honour not to return the full amount. They are like playground bullies who would rinse you for a piece of string rather than leave you unmolested.

One tenant, in London, had been waiting six weeks for his deposit, despite having caused no damage. The reason given by the letting agent was that the landlord was abroad and "quite a difficult person". Another had a third of her amount withheld for gardening costs, despite the fact that there was no evidence of damage, and nothing at all to suggest that the landlord had been left out of pocket, which is the criterion for failing to return a deposit.

Asked for something as simple as photographic proof, the agent replied: "We have no proof, but I don't believe we would have rented it out like that." When you ask as a journalist, however, these difficulties and infractions evaporate, and the money materialises; it's almost as if the objections were completely cooked up in the first place. I am struck by the bald impunity of it, the shoulder-shrugging certainty that the boot is never going to change feet.

There is something wrong with this market, and that is because it is asymmetrical: the returns for the landlord are massive. Every year of the past 18, money put into a buy-to-let mortgage has returned an average of 16.3% – all the way through a recession, immune to the slings and arrows hitting every other asset class. In the private rented sector, a third of homes are classed as non-decent (PDF). Whichever way you cut it, those landlords are rapacious – and there are plenty of them. Average rents in England and Wales will reach £765 a month by May and £800 by this time next year. It pleases the Residential Landlords Association (RLA) to compare rent costs to CPI inflation, but the salient comparison is with wages, which have been stagnant or falling for over five years.

Those brilliant returns for landlords are not free; or in other words, they are not windfalls from a beneficent universe. They are the direct result of a market in which people are being screwed for more and more of their income in rent. They cannot, therefore, amass the capital to buy, and so have to rent for longer, leaving them even more powerless in the face of price increases that bear no relation to their income. It's a racket, in other words: a game of Monopoly that won't end.

I wouldn't be moved to point out such blindingly obvious things were it not for a report published yesterday by the RLA that actually argues for less regulation. Regulation, incidentally, has been mainly responsible for the only improvements that have occurred within the sector. Since the introduction of the Tenancy Deposit Protection (TDP) scheme, in 2007, the percentage of tenants contesting withheld deposits has fallen from 40% to 1%. In the study written by Michael Ball of the University of Reading's Henley Business School, this very fact – that the number of unfairly held deposits is now pretty low – is used to argue that the scheme is a waste of money.

Ball concludes that regulation is too expensive, and has poor returns. The rationale is that the worst landlords are "unfazed" by punishment, while the best of them pick up the cost. This argument is typical of the way people with money are treated in economic equations (consider the similarities with the debate around higher rate taxation). Having agency, their likely behaviour is considered salient when decisions are made about the rules by which they should be bound. If they are likely to be unscrupulous, this militates against requiring scruples of them. Elsewhere in society the existence of dishonest people is used as an argument for more regulation, not less.

The study contends, too, that a shortage of housing equates to a "shortage of investment", which must be tackled by removing hurdles for people who want to invest. It's cynical and absurd. There is a shortage precisely because the only people investing are doing so for returns, and very few are investing for the sake of housing people.

These aims are at odds: the profit motive is driving up rents and driving down standards. Only 15% of houses in the social sector fail to meet the minimum standards for decency.

Even the success of the TDP scheme has to be put in a wider context. As Channel 4 reported earlier in the year, a massive loophole is afforded by the fact that there is no registered body for landlords; they can withhold a deposit, be reprimanded by the scheme, and simply set up as another company.

Words such as daft, unscrupulous and dishonest are diversions. The problem here is not with the personalities of landlords or letting agents but with the fact that they have too much power.

The tenants' group Generation Rent says landlords should be registered, and argues for more secure tenure. At the moment, letting agents advise setting one-year contracts because this makes it easier to ratchet the rent up. Generation Rent also points out that the poorest are also the most likely to have bad landlords, because they are the least able to front the cost of moving. With one party able to evict at any time, the power imbalance becomes absurd. It is no surprise that a fifth of students live in properties infested with vermin.

In the face of an urgent need for regulation, landlords argue for less. Nothing makes plainer their overweening power than this report, the audacity with which they tell you black is white.

Twitter: @zoesqwilliams