George Osborne's hopes of injecting fresh competition into high street banking were dashed on Wednesday after the Co-operative Group pulled out of talks to buy 632 branches from Lloyds Banking Group.

After more than a year of discussions, the biggest mutual in the country walked away from a deal that would have created a 974-strong branch network – three times the current size of the Co-op bank – and increased its share of current accounts from 2% to 7%.

It is a damaging blow the Co-op, which had boasted it would become a major challenger in high street banking. The Manchester-based group immediately faced speculation, strongly denied, that it would abandon its 150 years of banking altogether.

The branches – which Lloyds had been selling under the codename Verde – must be spun off by November under instruction from Europe as a consequence of Lloyds' £20bn taxpayer bailoutin 2008. A flotation under the TSB banner, a brand which disappeared from high street after Lloyds took over the bank in the 1990s, is now expected early next year. However, the 632 branches will be rebranded with a new blue TSB logo on the high street from September.

Only two months ago the chancellor had heralded the Co-op deal as helping to create "upstart challengers offering new and better services that shake up the established players". The Treasury had worked hard behind the scenes to try to smooth a transaction that was intended to give the Co-op an extra 4.8 million customers as a ready-made rival to take on the "big four" of Lloyds, Royal Bank of Scotland, HSBC and Barclays, although there was repeated speculation that City regulators had presented raised questions about the tie-up.

Richard Lloyd, executive director of the consumer body Which?, said the collapse was a "setback" for the government's attempt to "tackle the unhealthy dominance of our biggest banks". The shadow financial secretary to the Treasury, Chris Leslie, said: "This is not only bad news for the chancellor but for customers too."

"We need more competition in the banking sector - including a greater role for mutuals - to give customers and businesses a better deal," Leslie said.

A stock market flotation could value the branches at around £1.5bn – double the price being paid by the Co-op. It is also increasingly likely that Royal Bank of Scotland, also under instruction from Europe to sell off branches in turn for its taxpayer bailout, will float off its 300 branches after talks with Santander broke down last year.

A Treasury spokesman described the failure of the Lloyds deal as a "commercial matter". "The government remains determined to promote greater competition in the banking sector in order to provide consumers with more choice," the Treasury spokesman said.

Labour's City minister at the time of the banking crisis, Lord Myners, blamed the government. "Government undermining of confidence in banking has destroyed this deal," Myners said, adding there was "little prospect" of RBS getting "decent value" for the branches it has to sell either.

António Horta-Osório, chief executive of Lloyds, was told on Tuesday night by the outgoing Co-op boss, Peter Marks, that the deal was off. In a terse statement, the Co-op said it was "not in the best interests" of its members, who in effect own the group. "This decision reflects the impact of the current economic environment, the worsened outlook for economic growth and the increasing regulatory requirements on the financial services sector in general," the Co-op said.

Cabinet minister Ken Clarke argued that the view on the economy could not have changed in two years. Telling BBC Radio 4's World at One the decision was "bewildering", Clarke added the Co-op had "wasted a great deal of time".Marks, who retires as chief executive next month, t insisted speculation that Co-op would abandon banking was wrong: "The Co-operative group [goes forward] under our clear strategy and, as part of that, we will continue to develop our bank for the long term, offering a real alternative on the high street with our strong, established brand and our reputation as a trusted financial services business." But the decision will ultimately lie with his successor Euan Sutherland, who joins from Kingfisher in a fortnight.

Horta-Osório said he was disappointed with the move. "However, we are well advanced in our plans to bring the Verde business to the UK high street during the summer through the TSB Bank, and will now proceed with the option to IPO the business, subject to the necessary approvals," he said.

."