British troops in southern Iraq were scrambled to the Iranian border after the abduction of five British hostages in May 2007, in a failed attempt to stop them being taken into Iran, the Guardian has learned.

The troops were sent to the border area north of Basra to intercept the kidnappers after receiving intelligence that they were heading to the frontier from Baghdad, but failed to find them. It is unclear whether the British unit arrived too late or went to a different crossing point along the 1,500km border.

British officials today refused to give details of the attempted rescue operation, describing the issue as "extremely sensitive", but a British journalist visiting the Iraq-Iran border a few months after the abduction was briefed on the operation by British officers who had taken part.

As the abortive rescue attempt was being launched, the five British hostages were being driven from Baghdad to the border by their kidnappers, the day after their abduction in the centre of the Iraqi capital. A year-long investigation by Guardian Films found that Iraqi intelligence trailed the abductors – who included members of the special Quds force of Iran's Revolutionary Guard – and their victims to a brickworks across the border from the Iranian town of Mehran, which the Quds force used as a base for its operations in Iraq. The hostages were seen being transferred from one set of vehicles to another at the brickworks and then driven off.

The Foreign Office has maintained that it has no evidence that the British hostages were taken across the border but the head of US central command, General David Petraeus has confirmed that the Britons – Peter Moore, Jason Swindlehurst, Jason Creswell, Alec Maclachlan and Alan McMenemy – spent some of their subsequent captivity inside Iran.

The abductions of the five men came at a time when many British officers in Basra believed they were being sucked into a proxy war with Iran. Their troops were locked in intense urban conflict with Shia militias, believed to have been trained in Iran, armed by Iran, and even supported by Iranian forces. Farsi communications were intercepted between mortar teams targeting British positions in central Basra. The kidnapping team who seized the five Britons at Iraq's finance ministry in May 2007 were also overheard speaking what some thought was Farsi.

By the summer of 2007, intelligence officials in the US-led multinational coalition in Iraq estimated that Iranian-backed insurgents were responsible for roughly half of the attacks on their forces.

So concerned were British military commanders in Basra about the support, notably arms and roadside bomb technology, that Iran was giving Shia militia elements in the southern Iraqi city that they set up a separate battle group on the border specifically to try to stop the supplies. Furthermore, British units in the region were told to expect abduction attempts by Iranian-backed groups and rehearsed counter-measures against such a threat.

In June 2007, less than a month after the abduction of the five British hostages in Baghdad, the Ministry of Defence said an Iranian-backed plot to kidnap British troops in Basra had been foiled. A Shia Iranian-backed "special group" – elements of the Jayish al-Mahdi, Moqtada al Sadr's Mahdi army – had planned to wear stolen British army uniforms and target a small military liaison team based at the provincial joint operations centre in Basra where British soldiers were mentoring Iraqi forces. The plot was thwarted when the liaison team was tipped off by a loyal Iraqi policeman. A military spokesman in Basra said at the time: "We have procedures to combat this particular threat and they are effective."

Robert Baer, a former CIA agent and author of The Devil We Know: Dealing with the New Iranian Superpower, said: "Hostage-taking for the Iranians has been a tool of their foreign policy. It's been completely ingrained into their mind that when faced with a threat, they take hostages. Hostages are a deterrent for Iran and it's a very successful deterrent so that we see in Iraq, we've seen in Lebanon, we've seen in the American embassy takeover, hostage-taking is a tool of Iranian foreign policy which has worked these last 30 years."

Only one of the five hostages survived their abduction in May 2007 – Peter Moore, an IT consultant from Lincoln, who was freed on 30 December. The bodies of three of his guards, Swindlehurst, Creswell and Maclachlan, were handed over to British officials in Baghdad last year. They had all been shot dead several months earlier. The fourth, Alan McMenemy, is believed to have been killed too, but his remains have yet to be returned.