British travellers who ended up out of pocket as a result of a backlog at HM Passport Office this summer, including those who were forced to pay premium rates for a fast-track service, should be paid compensation, the Commons home affairs select committee has said.

The MPs said severe management failings at the top of the Whitehall arms-length agency meant it should now be brought back into the Home Office under the direct control of ministers.

The passport agency struggled to cope with issuing and renewing passports when its backlog of unprocessed applications spiralled past 550,000 in June, the period of peak demand. The MPs said the backlog left some British citizens unable to go on holiday, and sick children unable to return to the UK. The passport agency recorded a surplus of £124m over the past two years and awarded its staff bonuses of £674,000.

Keith Vaz, the committee chairman, said emergency measures ordered by Home Office ministers proved to be too little, too late. He said there had been a complete failure of management at the agency, whose leaders would be "unlikely to survive to the final round of The Apprentice".

"HM Passport Office should lose its agency status and be brought back under direct ministerial control following this appalling series of failures," Vaz said. "They have delivered a shamefully poor service to the estimated 5.6 million British citizens living abroad.

"The decision to transfer responsibility for processing overseas passport applications from the Foreign Office to the Passport Office was a mistake. The focus has been on departmental budgets rather than on providing a service paid for by citizens."

The MPs said they were concerned in particular that a number of people had ended up out of pocket. Some had to cancel their travel plans completely, some had to rebook flights at a late stage, and some incurred extra costs by travelling to passport offices in person to collect their delayed passports.

The MPs said people who paid for a fast-track upgrade if their passports were still stuck in the backlog when they were due to travel within seven days should also be compensated.

The payments should cover all those who made an initial application on or after 1 May, who subsequently upgraded to the fast-track service and who met the criteria for a free upgrade that was offered later.

The free upgrade was announced by the home secretary, Theresa May, on 12 June, covering all travellers who could provide proof that they were booked to travel within seven days and had been waiting for a new or renewed passport for more than three weeks. More than 16,000 people made use of the measure.

James Brokenshire, the immigration minister, said the Passport Office faced unprecedented demand in 2014, with more than 5m passports issued so far this year. "In response to this significant increase in demand, the home secretary introduced a series of measures to ensure that passports could be received by people in time for them to travel on their summer holidays," he said.

"This action has had a significant impact, reducing HMPO's outstanding number of applications from a peak of nearly 550,000 in June to around 90,000 today.

"Clearly this is little comfort for those who experienced delays and we need to make sure there is no repeat of the problems experienced this year."

Brokenshire added: "That is why the home secretary commissioned two reviews of HMPO to ensure it is working as effectively as possible. We are currently considering the findings of these reviews and will be announcing our response shortly."