After Ed Balls threatened to withdraw Labour's support for High Speed 2 and suggested that the £42.6bn allocated to the project could potentially be better spent elsewhere, I spoke to Andrew Adonis, the former transport secretary and the architect of HS2, at a New Statesman fringe event last night to get his response.

In his first reported comments since Balls's speech, Adonis, who is the head of Labour's growth review and shadow infrastructure minister, told me that the shadow chancellor had "raised the bar" for the project and that the "incompetent" coalition needed to demonstrate that it could "keep costs under control" if HS2 was to survive. He criticised the government's failure to pass legislation more quickly and to manage the programme effectively: "all they've done since coming to office is add £10bn to it".

Adonis, who warned in a recent New Statesman piece that it would be an "act of national self-mutilation" to cancel HS2, told me: "the current contingency fund of £14bn is too large and the cost needs to come down when the bill has its second reading in February/March." He added: "it's no surprise opinion is turning against it if people fear it will end up costing £100bn."

In his speech, Balls said: "the question is – not just whether a new high-speed line is a good idea or a bad idea, but whether it is the best way to spend £50bn for the future of our country. In tough times it's even more important that all our policies and commitments are properly costed and funded."

Adonis warned in his piece that the urgent need to increase rail capacity (the West Coast Main Line will be full by 2024) meant there was "no free lunch - or pot of gold which can be diverted to other projects in anything but the very short-term, with more costly consequences thereafter". But at a fringe meeting last night, Balls openly speculated on whether the HS2 money would be better spent on "building new homes or new schools or new hospitals".

With Miliband due to pledge in his speech today to build a million new homes over the course of the next parliament, it's unsurprising that Ballls is attracted by the option of scrapping HS2. It would allow Labour to raise billions for other projects while remaining within George Osborne's fiscal envelope. For Balls, understandably wary of making the case for borrowing to invest, that is political gold.