People caught up in the passport backlog should consider holidaying in Britain instead of going abroad, the tourism minister, Helen Grant, has said.

Speaking after the home secretary, Theresa May, issued an apology to people whose travel plans had been disrupted by the delays, she insisted the backlog of applications would be cleared. But she ran the risk of further enraging those caught in it by suggesting they have a "staycation".

Some called Grant's comments "appalling" and accused her of "not living in the real world".

Grant, who is at the World Cup in Brazil in her other capacity as sports minister, said she was very confident that applicants would receive their passports.

"But if they don't want to go away, we have some fantastic places to visit and holiday not that far from here. I think there's a lot to be said for the 'staycation'. People need to do what they like," she said.

She added: "I think we are going to have a great summer, we are certainly going to have a great summer of sport, too, there's lots of opportunities to build your holiday around a spoilt-for-choice list of events, the Tour de France Grand Départ, we've got the golf, we've got the Commonwealth Games, we've got football that we can watch on the TV.

"It's a wonderful place to have your holiday," she told the House magazine.

But she was criticised by 68-year-old retired nurse Eileen Shepherd, who had to miss out on a £1,750 cruise around the British Isles after being told the day before she was due to set sail that her passport, which she was required to produce in order to board, was not ready.

"How dare [Grant] tell us where to go on holiday? Cheeky sod," said Shepherd, from County Durham. She added: "I think people will be gunning for her. That comment is not helpful and not what we expect from elected personnel. She is not living in the real world with people like us. It is appalling. I've worked all my life and saved all my life to be able to go on one big holiday a year and I was robbed of that chance."

Martin Cook, 43, from Ipswich, said Grant's comments were "laughable, it is ludicrous". He said he endured delays getting his passport and only finally got it the day before he flew out to Prague for a weekend away with his wife, Annabel, last month, despite submitting his application four weeks earlier.

The BT project manager said of Grant: "If somebody has made their plans and put their passport application in in good time they should be able to travel where they like. It is a basic liberty."

Grant's comments followed the admission by the head of the Passport Office, Paul Pugh, that there were currently 480,000 applications being dealt with. HM Passport Office said it had been dealing with the highest demand in 12 years, with the summer surge in applications starting earlier than usual.

The government initially denied there was a backlog. But pictures printed by the Guardian showed boxes and boxes of applications stacked in a room at the Passport Office. The prime minister later admitted there was, in fact, a backlog, which both he and Grant have put at 30,000 applications.

David Cameron's official spokesman said: "It is always a matter for individuals and individual families where they may choose to take a holiday.

"The prime minister himself and his family, of course, have often holidayed in the UK. But the decisions individual families take are up to them."

Grant was embarrassed soon after being appointed sports minister when she fluffed every question in a sports quiz, failing to name the FA Cup winners or the current Wimbledon women's champion.

The Maidstone MP also came in for criticism recently after suggesting British girls might want to take up "feminine" activities such as cheerleading and ballet.