A British diplomatic effort to reach out to Libyan rebels has ended in humiliation as a team of British special forces and intelligence agents left Benghazi after being briefly detained.

The six SAS troops and two MI6 officers were seized by Libyan rebels in the eastern part of the country after arriving by helicopter four days ago. They left on HMS Cumberland, the frigate that had docked in Benghazi to evacuate British and other EU nationals as Libya lurched deeper into conflict. The diplomatic team's departure marked a perfunctory end to a bizarre and botched venture.

"I can confirm that a small British diplomatic team has been in Benghazi," said William Hague, the foreign secretary. "The team went to Libya to initiate contacts with the opposition. They experienced difficulties, which have now been satisfactorily resolved. They have now left Libya."

Audio of a telephone conversation between the UK's ambassador to Libya, Richard Northern, and a senior rebel leader was later leaked.

Northern suggested in the call that the SAS team had been detained due to a misunderstanding.

The rebel leader responded: "They made a big mistake, coming with a helicopter in an open area."

Northern said: "I didn't know how they were coming."

Despite the failure of the mission, Hague indicated that Britain would continue to try to make contact with the opposition.

"We intend, in consultation with the opposition, to send a further team to strengthen our dialogue in due course," he said. "This diplomatic effort is part of the UK's wider work on Libya, including our ongoing humanitarian support. We continue to press for Gaddafi to step down and we will work with the international community to support the legitimate ambitions of the Libyan people."

According to Guardian sources, the British intelligence and special forces unit were caught near the al-Khadra Farm Company, 18 miles (30km) south-west of Benghazi. A senior member of Benghazi's revolutionary council said: "They were carrying espionage equipment, reconnaissance equipment, multiple passports and weapons. This is no way to conduct yourself during an uprising.

"Gaddafi is bringing in thousands of mercenaries to kill us, most are using foreign passports and how do we know who these people are?

"They say they're British nationals and some of the passports they have are British. But the Israelis used British passports to kill that man in Dubai last year."

Rebel leaders said claimed the captives had been treated well and would be released as soon as the British government vouched for their identity with the rebel command.

The news follows Sunday Times claims that an SAS unit was being held by rebel forces it had approached in an attempt to open up diplomatic channels to opponents of Muammar Gaddafi.

Whitehall sources said on Friday it needed to learn more about the leadership of the anti-Gaddafi forces and find out what logistical support they needed, but would not give arms to the rebels, as an international arms embargo was in place.

British officials during the day declined to comment on reports that special forces were being held but defended the objective of the mission.

The defence secretary, Liam Fox said: "It is a very difficult situation to be able to understand in detail. There are a number of different opposition groups to Colonel Gaddafi in Libya who do seem relatively disparate. We want to clearly understand what the dynamic is here because we want to be able to work with them to ensure the demise of the Gaddafi regime, to see a transition to greater stability in Libya and ultimately to more representative government.

"So getting a picture of that is relatively difficult, as is widely reported. Communications are being interrupted, there are difficulties with mobile phones, with the internet potentially being interfered with.

"So we are trying to build a picture – it's essential that the government does that and it's essential that all western governments do that so we are able to get a clearer idea of what we are able to do in terms of helping the people of Libya."

David Cameron, speaking at the Tory party spring conference in Cardiff, repeated his call for "Gaddafi to go". "On Libya, our strategy is clear," he said. "We will continue to intensify pressure on the regime. We will continue to state clearly that international justice has a long reach and a long memory, and that those who commit crimes against humanity will not go unpunished. We will continue to provide humanitarian assistance to those affected by this crisis, and continue to demand access for aid agencies to reach those in need.

And we will continue to plan, with our allies, for every eventuality. "

The Sunday Times reported Libyan and British sources confirming the SAS unit had been detained by rebel forces it had approached to secure a meeting with a junior diplomat to offer help in their fight against Gaddafi. The mission backfired when rebel leaders in Benghazi objected to foreign interference from governments which had not yet formally recognised them as Libya's legitimate rulers, it said.