The number of people employed by the government to work on the UK's response to the effects of climate change has been cut from 38 officials to just six, triggering accusations that David Cameron's promise to be the greenest government has been abandoned.

The UK is facing a multi-billion pound bill over the next few years for the costs of adapting to the effects of climate change – including flooding, much fiercer storms, droughts, heatwaves and more extreme weather. The government's advisers, the Committee on Climate Change, have warned that the measures needed to prepare the UK's infrastructure will include defences for power stations, transport and communication networks, changes to how buildings are constructed, and new ways of trying to prevent flooding, such as an upgrade to the Thames Barrier.

But the number of officials charged with dealing with the issue within the Department for Environment, Food and Rural Affairs (Defra) has been dramatically reduced. A freedom of information response to a question by Friends of the Earth confirmed the reduction from 38 to six posts.

Defra insisted that the changes were the result of a restructuring. A spokesman said: "How we adapt to any impacts of climate change has been embedded into every policy team in Defra. Staff numbers on the adaptation team will therefore be reduced and the expertise moved to other parts of the department. A larger team will then come together to deliver the next climate change risk assessment in 2017."

But the staff reduction was condemned by one of the former most senior Whitehall officials on climate change. John Ashton was charged with leading the UK's diplomatic efforts to forge a new international agreement on cutting greenhouse gas emissions at the United Nations.

In a fiery speech at the Royal Society of Arts on Thursday night, in which he lambasted his former political masters from every party for their failure to get to grips with the urgent problem of global warming, Ashton attacked ministers for "spooking potential investors" in the UK's energy infrastructure.

He said Cameron's famous husky – referring to when the then opposition leader was photographed in the Arctic as part of his efforts to change the image of the Conservatives – was dead "with a neat hole between its eyes".

Ashton, now a director at the environmental thinktank E3G, also called for a target for the decarbonisation of the UK's electricity industry to be included in the forthcoming energy bill, which is currently going through parliament.

The proposed target of making the UK's electricity generation almost carbon-free by 2030 has been removed by the government at the insistence of the chancellor, George Osborne, who has quashed all proposed environmental targets beyond 2020. But the former Tory minister Tim Yeo is planning a rebellion on the bill when it comes before the Commons next month.

Ashton said: "It is vital that the target be restored. I can't myself see how any MP who votes against the target will thereafter be able credibly to claim that they support an effective response to climate change."

He said the UK could not move out of recession without looking to green industries: "The government may think it has a growth story. Nobody else does. If it would stop looking at low-carbon growth in the way the Spanish inquisition looked at heretics, it could find one in front of its nose. The economy as a whole bumps along the bottom. The low-carbon economy keeps growing at nearly 4% [a year]."

Andy Atkins, director of Friends of the Earth which co-ordinated the event at which Ashton spoke, said: "After a year that has already brought flooding and other extreme weather to the UK, it's shocking that the department responsible for protecting us against the effects of climate change is to pare its staff to the bone."