Peer did not want to cause government any further embarrassment, Downing Street sources say

The Conservatives suffered their first embarrassing resignation since the election today when Lord Young quit as enterprise adviser to David Cameron.

Young caused controversy by saying Britons had "never had it so good" in this "so-called recession", prompting frustration in No 10 and provoking a welter of criticism from Labour.

Downing Street sources said Young had been upset by the furore and had thought about his position overnight. He did not want to cause the government any future embarrassment, so he decided to resign.

Cameron was grateful for the good work the peer had done for him, the sources said.

The prime minister initially appeared simply to rap the peer over the knuckles, condemning the comments and saying that Young would be doing a "little less speaking" in the future.

But with criticism mounting, Young offered his resignation which was rapidly accepted. There were questions about why the peer was not asked to resign sooner, allowing speculation about his gaffe to continue.

The fact that Lord Young is a multi-millionaire and former trade minister from the Thatcher era threatened to contaminate Cameron's brand of new Conservatism.

Labour claimed that the whole saga proved that the Tories are out of touch with people. Ed Miliband said: "Lord Young is right to go; I think his remarks are frankly disgraceful and many of the people who are struggling up and down this country with the consequences of the recession that we had, the consequences of the spending cuts that we are seeing, will be insulted by his comments. I do have to say though that it reflects very badly on the prime minister that he made this appointment of someone who seems so out of touch with people up and down this country.

"Part of what this government has to do is demonstrate they do understand the consequences of the decisions they are making and the way people are struggling every day in our economy. I'm worried that people like Lord Young indicate a government that doesn't understand that."

Young was drafted in to advise Cameron, partly as a sop to the traditional right in the party who remember him as an effective, though sometimes unremarkable, minister in the Thatcher government. He carried out a review of health and safety laws for the coalition promising a new "common sense" approach to tackling a growing compensation culture.

On Wednesday night he was awarded "peer of the year" at the Spectator awards by the prime minister.

But today's Daily Telegraph reported Lord Young saying: "For the vast majority of people in the country today, they have never had it so good ever since this recession – this so-called recession – started, because anybody, most people with a mortgage who were paying a lot of money each month, suddenly started paying very little each month.

"That could make three, four, five, six hundred pounds a month difference, free of tax. That is why the retail sales have kept very good all the way through."

Labour frontbenchers Angela Eagle and John Denham had called for the peer to stand down. Denham had called his remarks "crass, insensitive and ignorant".

The prime minister's spokesman was repeatedly asked at today's Downing Street media briefing if Cameron retained full confidence in Young, but refused to say yes, telling reporters: "You are asking a question that you would normally ask about a member of the government. It's a different role."

He said the adviser had produced a good report on health and safety and added that he was now working on a study of government relationships with small business, to be completed within a year. "We have not said what he will do after this," added the spokesman.

The spokesman added that Young's remarks were inaccurate since they underestimated the impact of a recession, which had created much uncertainty.

Earlier, No 10 had made it clear that Cameron was "deeply unimpressed" by Young's comments, made during an interview in the Roux restaurant in Westminster.

Young wrote to Cameron last night to apologise and express his "profound regrets". He described his comments as insensitive, adding: "I am not a member of the government and played no part in the spending review. I deeply regret the comments and I entirely understand the offence they will cause.

"I should have chosen my words more carefully. Low mortgage rates may have eased the burden for some families in this country. But millions of families face very difficult and anxious times."

Young's letter to the prime minister said: "I have considered my position overnight and in view of the reaction to the reporting of the interview I gave earlier this week feel that it would be right to resign forthwith from my position as your adviser. I am a very strong supporter of you and the coalition government and for what you are about to achieve. It was always my ambition to play a small part in helping you restore the fortunes of our nation and you should know that you will continue to have my support in every way.

"Finally may I say how much I appreciate the way in which you and your government has made a start on the considerable task ahead and wish you the very best of good fortune."

Tony Woodley, the joint general secretary of the Unite union, said Young had "let the mask slip". He went on: "His Thatcherite claptrap shows that this country has passed into the hands of an out-of-touch, unaccountable elite. But Lord Young has done the people of this county a favour – now we know exactly what this government thinks of them."

But Lord Jones, the ex-CBI chief who was a minister in Gordon Brown's government, said Young had been "partly, in one aspect of the economy, right". He told BBC Radio 4's Today programme: "He was generalising and the media pick up on a generalisation and it suddenly becomes true or false. It is true that if you went into late 2007, were on a tracker mortgage and you were in a sustainable job, then the next 18 months, you had disposable income and, if you forgive the analogy, you had a good recession.

"He was partly, in one aspect of the economy, right. On the other hand if you were on short time or if you were out of work and if you were a pensioner or another person relying on your savings, then the last thing in the world you want are low interest rates, you had a very bad recession. You can't generalise at all."

Accepting the Spectator award for peer of the year on Wednesday night Lord Young said: "I think this is tremendous but what I enjoyed even more was the email I got that said that the Spectator for 25 years has had a tremendous record of picking future prime ministers and future stars, and I thought well 'this is great'.

"But then I realised that all my colleagues I get on very well with, they regard me with an air of benevolence, [as] someone who survived from the dim and mythical ages of Thatcher, and if they thought I had a future ... then [they] might get worried so thank you very much indeed."