More than £400m diverted from supporting communities in need across the UK to help fund the London 2012 Olympics has yet to be returned, and the shortfall has raised questions about the Games’ legacy.

With the 2016 Rio Olympics now underway, there are calls for the new mayor of London and the prime minister to ensure the cash – taken from the Big Lottery Fund which makes small grants to local charities – is paid back.

The previous mayor of London, Boris Johnson, stated that a key legacy aim of the 2012 Games was the “convergence in life chances and prosperity between east and west London, and to bring the city and its communities together”. But charities say the failure to return the lottery cash has undermined this commitment.

Ciaran Price, policy officer with the Directory of Social Change, the organisation that campaigns to improve conditions for charities in the UK and is leading calls for the return of the money, said: “The government failed to honour its promise to Britain’s communities, which have been struggling in the wake of a devastating recession, to repay the £425m it took from the Big Lottery Fund to construct the Olympic Village.

“While many charities are seeing demand for their services rising at a speed never before experienced, and while they are finding it more and more difficult to get the financial support they need to meet this new level of demand, the government has been sitting on this money, continuously trying to kick it further into the long grass, hoping we’ll somehow forget about it.”

It emerged after the Games that the government had underspent on the Olympics by more than £500m but, despite this, none of the £425m taken from the Big Lottery Fund has been returned to good causes.

Charities were further dismayed when the Olympic stadium, which cost £486m to build, was handed over to the Premier League football club West Ham on a 99-year lease, at a cost to the club of £2.5m a year. The Directory of Social Change has now written to both mayor of London Sadiq Khan and the prime minister, calling on them to intervene.

The organisation says it is vital that lottery money goes to the charitable causes for which it is intended. In its letter to Theresa May, it points out that when she was shadow secretary for culture, media and sport she stated: “We believe that the lottery should be independent of government but accountable to parliament. It should not become a tool of government.”

In a speech to parliament in 2005, May claimed that governments should not be able to raid lottery funds. “Every pound that the government choose to snaffle that way is a pound that cannot go to help community groups or to preserve our historic buildings.”

Debra Allcock Tyler, chief executive of the Directory of Social Change, claims in the letters that more than 10,000 charities could be supported if the money were returned, and around eight million people would benefit.

She acknowledges that the government has committed to repay the money via the sale of assets in the Olympic park. But the previous and current governments have said this will not happen until the next decade at the earliest, and possibly the one after.

“This is too late for so many communities in need,” Allcock Tyler says in her letter to Khan. “We maintain that giving back this money now would make a huge difference to organisations and the individuals they serve across London and the rest of the UK at a time when so many people are in need of support.”

The Directory of Social Change claims that the Brexit vote has given a renewed urgency to claiming back the millions owed to the Big Lottery Fund. It points out that the EU currently gives £200m to 249 charities in the UK and that the future of this funding is now uncertain.

“Now, in a period of unprecedented uncertainty in the wake of the EU referendum result, the money is needed more than ever,” Price said. “Charities are struggling to cope. The decision to give away the Olympic football stadium to West Ham United for the next century, for the private benefit of its wealthy club owners, casts serious doubt over the government’s sincerity.

“In reality, £425m is such a small amount of money and it would have such a huge, positive impact for Britain’s communities, which will improve the lives of absolutely every person in the country. The government cannot deny the sector this money any longer.”