Small businesses have responded furiously to claims by the Culture Secretary, Jeremy Hunt, that the Olympics were "a very good period" for tourism.

Travel organisations and shopkeepers around London and many working in tourism outside the capital described the period of the Games as one of their worst and questioned how easy it would be for many small firms to recover.

They were angered yesterday when Mr Hunt denied there had been a fall in trade and went on to claim the Games had been good for the industry. Mr Hunt told The Independent: "It was quieter in the first week of the Olympics, but picked up a lot in the second week. West End businesses did well – theatre bookings up 25 per cent on a year ago according to Andrew Lloyd Webber, restaurant bookings up 20 per cent according to Visa."

But Neil Wootton, managing director of the sightseeing specialist Premium Tours, said business was down by 42 per cent year on year: "It will take a long time to repair the shortfall of this summer. The knock-on implication has been felt by all attractions, venues, hotels and pubs we use – with some privately owned establishments calling us in panic-stricken attempts to drum up business. The biggest concern is how many smaller companies, which depend upon the key selling months of June, July and August, can survive the winter."

JacTravel, a hotel wholesaler, reported that London bookings were down by more than one-third – in contrast with a 45 per cent rise in sales in key Continental cities. A spokesman said: "There was a clear displacement of tourists who would normally come to London, though UK domestic tourists began to appear after word got out that London was deserted and there were incredible bargains to be had."

A West End art dealer with a high proportion of overseas customers, Rosslyn Glassman, said: "Turnover has been half that of regular weeks."

He said official warnings to avoid the capital had been far too strident.

The Culture Secretary rejected the criticism. "What we actually had last week was record numbers travelling on the Tube – 4.61 million people on certain days. We got everyone to their Olympic events on time. We wouldn't have been able to do that if we hadn't warned people that central London was going to be busy, discouraging some non-essential travel."

Figures released by Heathrow's owner, BAA, revealed far fewer arrivals than expected for the Olympics. The company had predicted that 26 July, the day before the Opening Ceremony, would be the busiest day in its history for arrivals, with a record 138,000 passengers touching down. The forecast was 36 per cent higher than the actual number of travellers. With just 102,000 arrivals, the day proved quieter than an average summer's Thursday at Europe's busiest airport.

A spokeswoman for BAA said: "We assumed passenger numbers would be at the upper end of estimates. We think this was the responsible and prudent thing to do, and it meant that we could be confident that our plans would be robust."

The shortfall in inbound visitors hit business elsewhere in Britain. Nick Brooks-Sykes of Bath Tourism Plus described the Olympics period as "quite difficult" for the city, with a drop of up to one-fifth in visitors.

Andrew Johnson, director of Camera Obscura on the Royal Mile in Edinburgh, said: "Our visitor numbers have fallen by 10 per cent over the past two weeks. That is actually rather good compared to the other attractions I have spoken to."

Neil Wootton of Premium Tours said the Government had inflated expectations of visitor numbers, which in turn had led to unrealistic hotel rates: "There was no need for normal leisure tourism to be turned away. The authorities had a responsibility to consult, advise and even lay down guidelines for how hoteliers structured their pricing during the Games."

The chief executive of Visit Britain, Sandie Dawe, said: "We always knew that in the year of the Olympics it would be quite a challenge to hold on to our regular tourism market. So far this year we're doing pretty well, we're two per cent up in the first six months. Of course that doesn't deal with the Olympic period, but longer term we think it has fantastic prospects. The world now sees Britain as a place that can party and let its hair down."

In a speech at Tate Modern on the South Bank, Mr Hunt unveiled a £10m plan to increase inbound tourism by one-third to 40 million by 2020. He said: "We've been at the centre of global attention in a way that has never happened before in our lifetimes and may never happen again. Let's turn that into people who actually want to come and visit us."

Case study: 'The Government told them to keep away. They did'

Tim Bryars is a dealer in antique maps in the West End and relies heavily on London's tourist trade

"Hosting the Olympics was a privilege. However, that should be set aside from the corporate nature of the organisation of the Games, and the asinine insistence by Mr Hunt and others (including Boris Johnson, who should know better) that the Games were good for business, a mantra they've clung to before, during and after the event".