When this column was the first to highlight the perils of buy-to-let – both social and economic – Gordon Brown was chancellor. When we first revealed the vast tax reliefs paid to landlords, Alistair Darling was about to assume the role. Why did it take a Tory chancellor – much to the shock of his voters – to take the first steps to clamp down on the excessive tax privileges enjoyed by landlords?

Stephen Ludlow, chairman of big letting agent Ludlow Thompson, is – like many in the industry – bewildered by George Osborne’s cuts to tax relief. “Many investors voted against Labour in the election because of their attitude towards buy-to-let. They will not have expected such a move from the Conservatives,” he said after the budget speech.

In the immediate aftermath of the speech the shares of specialist landlord lenders and housebuilders fell sharply, although most have since recovered. They underestimate the impact Osborne’s tax bombshell is going to have. As we report today, investors who rely on a typical buy-to-let mortgage to finance a purchase will see their returns shrivel dramatically. Annual returns could be slashed by around two-thirds, leaving investors with yields of as little as 0.6% a year on their properties. If and when interest rates rise, it could be game over for the amateur investor.

How much have we read about newly liberated pension savers cashing in to invest in property? If they have to pay 6%-7% for the loan, can’t automatically write off 10% of their interest payments as “wear and tear” (what a fiddle that was), can only obtain tax relief at 20%, and can’t rely on the taxpayer to subsidise them with housing benefit payments, then it’s difficult to see how the financial equation will stack up.

Property investment will pay out if house prices continue to rise, with investors cashing in on capital gains, subject of course to capital gains tax. But it’s a brave investor who thinks that at today’s elevated level of house prices there is going to be much more in the way of price growth. The market is at a stalemate: on the one hand prices are supported by a rising population, low interest rates and persistently low building volumes. On the other, mortgages are more difficult to obtain and, more importantly, income multiples at near breaking point in much of the country.

We will hear heaps of tosh about how Osborne’s attack on landlords will hurt tenants most. The supply of rented property will dry up, they say, in the deluded belief that families today choose to rent rather than being forced to by high house prices. In any case, there are indications that the new tax regime will support institutional investment in rental property, which if it happens should lead to a saner, rather more German-style lettings market.

• The silent losers from this week’s budget are the self-employed. Not the high-earning IT and financial consultants, but the millions of people who have been forced into long hours and miserable pay since the financial crisis struck. For them the new national living wage is meaningless. There will be no automatic pay rise, but there will be a cut in tax credits, particularly if they had so irresponsibly had a third child.

Self-employment has grown dramatically over the past decade and now accounts for one in seven of the workforce. Their pay is, on average, 40% below that of conventional employees, according to the Resolution Foundation. Between a quarter and a third of the growth in the self-employed is down to an increase in the numbers moving from unemployment into self-employment rather than becoming employees.

Meanwhile, employers will do their best to escape the new minimum wage legislation by outsourcing work to the “self-employed” when in truth they will be disguised employees. We should welcome the higher minimum wage – though it falls well short of what is needed to counter cuts in tax credits. And if the new living wage is going to have any credibility, enforcing it thoroughly is crucial.