Conservative councils will build on average 20 social homes for the poorest people in society by the end of the parliament – and in the housing minister's own back yard, just one house a year will be built until 2015, according to a survey.

Freedom of information requests by the Labour party to 324 local councils showed that Tory local authorities were building fewer than half as many social homes as Liberal Democrat councils and fewer than a fifth of those in Labour authorities, which were planning an average of 100 properties for the poorest over the next three years. Of the 324 councils, 246 responded to the Labour request.

Concern exists that the number of social homes, be they in council hands or built by housing associations, has collapsed under the coalition. Last year there was a 91% fall, from 35,690 to 3,305, in the number of social homes, where rents are roughly a third of market rates, being built.

In the 50 square miles of Tory-run Welwyn Hatfield, the constituency of the Conservative housing minister, Grant Shapps, only three social homes are being built in the next three years. In the 25 London boroughs that responded to the study, 4,000 social homes will be built between now and 2015.

Further homes may be built, but these either lack planning permission or funding. Almost 5 million people are now on waiting lists for social housing; the results of the survey reflect in part the coalition's decision to slash the capital grant for social housing by 63% in 2010.

Hilary Benn, the shadow communities secretary, whose office conducted the study, said: "These figures couldn't be clearer: Labour councils are building many more social homes than Tory councils, showing what a difference it makes having Labour in government locally.

"By choosing to make social housing a priority, Labour is providing much-needed homes, creating jobs and boosting local economies. Ministers keep telling us we have a housing crisis, but their Tory colleagues are simply failing to build the homes that people need, at a rent they can afford."

Among Labour councils, Islington plans to build 708 social homes by 2015, Leeds 387 and Birmingham 338.

The business secretary, Vince Cable, said earlier this year that economic recovery "requires a big expansion in social and private housebuilding".

Housing experts believe a key problem is that developers bought much of their land at the height of the property boom and therefore need higher rents that few people seeking social housing can afford.

This commercial reality has seen Shapps, a rising Conservative star who has been tipped for promotion to the cabinet, champion the need to build "affordable housing", for which rents are roughly double those in social housing. Higher rents allow developers to recoup their investment sooner, and lure City investors to finance building of new homes.

With "affordable rent" set at up to 80% of local market rents for new tenants, Shapps argues that there will be enough cash to build 170,000 homes by 2015. Last year only 12,000 affordable properties, for rent or shared ownership, were built.

In a statement, Shapps said: "This government is delivering £19.5bn of funding to build 170,000 more affordable homes across the whole of England from 2011 to 2015. Contracts have been signed for building in council areas of all political colours, including 55,000 in London being delivered by a Conservative mayor.

"This survey is completely bogus and yet another dodgy dossier from Labour."

A spokesperson for Welwyn Hatfield council said: "As the freedom of information request only applied to 'social rent' and also only to those properties that have received all necessary approvals, including therefore planning permission, it is not surprising that this is a low figure … there will be many schemes that are likely to be delivered within the next three years which won't yet have all necessary approvals."