Build on your land or we’ll take it ourselves. That’s the tough-talking message that Britain’s opposition Labour Party is sending to construction companies they suspect of exacerbating the country’s housing crisis.

In a recent speech, Labour leader Ed Milliband promised that if elected, his party would give local authorities the right to fine developers who sit on unused land while waiting for its value to rise, rather than build on it. In extreme cases, he would even grant compulsory purchase powers to forcibly take the land out of the owners’ idle hands.

The figures Milliband has cited for these radical measures are certainly sobering. Across the UK, planning permission has been granted for 400,000 homes that haven’t been built. The number of completed dwellings is at a 50-year low – at least – at just over 135,000 in 2012-2013, compared to 330,000 dwellings 40 years ago. In London especially, housing costs are skyrocketing, with the average London home price projected to smash through the £500,000 ceiling by 2018. Despite this, 45 percent of the city’s plots with planning permission are in the hands of non-developers. In a country that has the fastest population growth in the European Union, this inertia is contributing to an increasingly desperate housing shortage.

Developers nonetheless insist that they’re not to blame for the housing shortage, and have questioned the logic of Milliband’s claims. Industry figures point out that the process of buying a site and gaining full planning consent can take at least three years, and insist that firms don’t have the capital to just sit on undeveloped land and wait for it to appreciate in value.“House builders buy land to build homes – they are not land companies,” says Peter Stacey, director of UK planning advisors Turley Associates. “We act for house builders across the country and have yet to see any evidence of land banking.”

It’s possible, however, that developers are defending themselves against an attack that’s not really aimed at them. According to a study commissioned by London’s mayor, London’s 45 percent chunk of permitted homes not currently being built are in the hands of “firms such as owner-occupiers, investment funds, historic land owners, government, and ‘developers’ who do not build.” In other words, while developers hold some plots, many of the owners are actually speculators with little connection to the construction industry.

This highlights a potential weak point in Labour’s rhetoric. If they really intend to fine and repossess, their targets would also include house owners who have bought up, say, the plot next door. And “historic landowners” could include the Crown Estate, a huge property portfolio owned by the queen (which has admittedly already taken steps to free up land for development). If they target small property owners, they risk being accused of bullying. If they ignore them, they stand vulnerable to charges of selectively bashing big business.

Furthermore, building more homes alone might not significantly remedy England’s housing crisis unless a different financing model is introduced. In the London area at least, little housing is available on the market partly because new developments rely on off-plan sales to investors (often international) for much of their financing. This means much stock is sold to large-scale buy-to-let investors before it is even complete.

The high costs of moving also limit the supply of real estate, as do owners profiting from the house-price boom by moving on mortgages secured on their existing home. Perhaps inevitably, considering the amount of money to be made, there is a trend of people retaining their homes as profitable rental properties rather than selling them on. Ignore these issues, and many of the UK’s housing problems will remain.

Given the political minefield that Labour would have to negotiate to implement their threat, it’s far from guaranteed that it will ever make it into law. The party has declared plans to get 200,000 new homes built in England every year, but crucially, at least a portion of those promised 200,000 homes would come from local governments building social housing – Britain’s high construction levels 40 years ago were substantially fueled by public investment. If Labour comes to power and acts on these promises, it will be remembered for generations. Its threat to fine and dispossess land profiteers, however, may well be forgotten as standard pre-election saber-rattling.