Police arrested six men over an alleged terrorist plot against the pope hours before he delivered one of his trip's key addresses to four former prime ministers and hundreds of parliamentarians and religious leaders in Westminster Hall.

Five London street cleaners were arrested at gunpoint by counter-terrorist officers in a dawn raid at the depot where they worked in Westminster, London. The sixth man was arrested on Friday afternoon in north London.

Searches by officers at up to 10 addresses were continuing, but no equipment linked to bomb-making or anything that could obviously be used to stage a terrorist attack was recovered.

Some of the men arrested are believed to be of Algerian heritage. They were being held and questioned by detectives who are also trying to establish their identities.

Security sources described the arrests as "precautionary" and Scotland Yard officers are understood to be bracing themselves for criticism if their suspicions are unfounded and the men are released.

News of the arrest came hours before Benedict arrived at Westminster Hall, where he delivered an address to several hundred of the most prominent people in British public life, among them Gordon Brown, Tony Blair, Sir John Major, Lady Thatcher, Nick Clegg and the archbishop of Canterbury, Rowan Williams.

One knowledgeable source said the arrests were made because of concerns including conversations overheard between some of those arrested, the fact their jobs gave them access to areas to be visited by the pope, and the difficulty in gauging the threat in a very short time.

Other sources said that the level of alarm in Whitehall was "low key". There was no meeting of the government's emergency committee, Cobra, and the terrorism threat level remained unchanged, indicating that there was no credible evidence pointing to an imminent attack.

Despite the scare, the 83-year-old pope continued with the second day of his four-day state visit. He was told of the arrests at his first engagement of the day – a visit to Britain's biggest Catholic teacher training establishment, St Mary's University College in Twickenham.

Father Federico Lombardi, the Vatican's head of press, said that the visit would carry on with "courage and joy" despite the arrests, adding: "We are calm … We are totally confident in the work of the police and Scotland Yard."

Scotland Yard said the policing plan had been reviewed, but the pope's itinerary would remain unchanged.

The six men – who are aged 26, 27, 29, 36, 40 and 50 – were arrested under the Terrorism Act 2000, on suspicion of the commission, preparation or instigation of acts of terrorism. Searches were carried out at their work and home addresses in north and east London. Other locations were searched later in the day with police seizing computers for evidence of terrorist planning or signs of extremism.

Westminster council said the first five to be arrested worked for Veolia Environmental Services, a contractor employing 650 on-street staff.

Sources said the men were not known to counter-terrorism officials.

The information that led to the arrests was not the result of intercepts or undercover work, but was, sources said, more akin to an overheard conversation that could be interpreted as posing a threat.

It came to police on Thursday as the pope prepared for a round of public events in London. His schedule was a key factor in the decision by senior officers to act and thereby quash any potential threat.

The timing of the arrests will expose Scotland Yard to criticism if the men are released without charge. However, the former head of counter-terrorism at Scotland Yard, Bob Quick, said police had little choice. "You don't have much time to evaluate the information, and you cannot take the risk," he said. "The duty on the police is to err on the side of caution, even if someone is not charged, rather than not acting and finding out you had a real plot which came to fruition."

Quick added there was a public misconception about the purpose of arrests in terrorism cases: "An arrest is a means of investigation, it does not mean someone is guilty of an offence."

Counter-terror sources said M15 was also investigating the men's background.

A huge security and public order operation swung into action on Thursday when the pope touched down in Britain. Thousands of officers are involved in the operation from forces including the Met, Strathclyde, Lothian and Borders, West Midlands and British transport police and the cost of policing the papal visit could reach £1.5m

Senior police officers said last week that there was no information to suggest any specific group wanted to attack the pope, but warned against underestimating the "passion and the fervour" the visit would evoke.

Police also interviewed mentally unstable people who they feared could pose a threat to the pope.

The pontiff faces a slightly less gruelling day on Saturday, beginning with a meeting with the prime minister and ending more than 12 hours later after a prayer vigil in Hyde Park.