Libya has demanded an explanation for the "kidnapping" of one of its citizens by American special forces, hours after a separate US military raid on a terrorist target in Somalia ended in apparent failure and retreat.

In Tripoli the US army's Delta force seized alleged al-Qaida leader Nazih Abdul-Hamed al-Ruqai, known by his alias Abu Anas al-Liby and wanted for the 1998 bombings of the US embassies in Kenya and Tanzania that killed more than 220 people.

The New York Times reported that Liby was being held in military custody and interrogated on board a navy ship, the USS Antonio, in the Mediterranean.

But US navy Seals suffered a major setback when they launched an amphibious assault to capture an Islamist militant leader said to be Ahmed Godane, described as Africa's most wanted man and the architect of last month's attack on the Westgate shopping mall in Kenya. The elite Seals were beaten back by heavy fire and apparently abandoned equipment that the Somali militants photographed and posted on the internet.

As dramatic details of Saturday's twin operations emerged, US secretary of state John Kerry insisted that terrorists "can run but they can't hide" , but faced growing questions about America's military reach in Africa and the consequences of unilateral aggression.

Speaking in Indonesia on Monday, Kerry said the seizure of Liby complied with US law, the Associated Press reported. He said the suspect was a "legal and appropriate target" for the US military and would face justice in court. It was important not to "sympathise" with wanted terrorists, Kerry said.

Liby was captured outside his family home at 6.15am in Noufle'een, a quiet suburb in eastern Tripoli, according to witnesses, but there were conflicting reports over who took him. His brother, Nabih, told the Associated Press that Liby was parking when a convoy of three vehicles encircled his car. Armed gunmen smashed the car's window and seized Liby's gun before grabbing him and taking him away, the report said. The brother said Liby's wife saw the kidnapping from her window and described the abductors as foreign-looking armed "commandos".

But Liby's son Abdullah insisted Libyan forces were involved. Appearing on Tripoli's Nabir TV station, he said: "The people who took my father were Libyan, not Americans – they spoke with Tripoli accents.

"My mother was listening to the voices in the street and could see it all through the window. There were two cars and a bus with blacked-out windows and no number plates."

He said his father was dragged from his car and arrested while it was still moving, and the vehicle, driverless, continued driving empty down the road.

Liby, who was thought to be a computer specialist for al-Qaida and lived in Manchester in the UK during the 1990s, is believed to be 49 and on the FBI's most-wanted list with a $5m (£3m) bounty on his head. Pentagon spokesman George Little said he is "currently lawfully detained by the US military in a secure location outside of Libya".

Liby was expected eventually to be sent to New York for criminal prosecution, the New York Times reported.

Libya's government refused to say whether its forces were involved in the arrest and claimed it had not been informed in advance. A statement from the prime minister, Ali Zaidan, said: "The Libyan government is following the news of the kidnapping of a Libyan citizen who is wanted by US authorities. The Libyan government has contacted US authorities to ask them to provide an explanation."

Thousands of miles away in Somalia, US special forces carried out a raid that was no less audacious but had a very different outcome. It was reportedly planned a week and a half ago in response to the Nairobi attack and came 20 years to the week after an American mission that infamously went awry when Somali fighters shot down two Black Hawk helicopters.

Members of Seal Team Six – the unit that killed al-Qaida leader Osama bin Laden in his Pakistan hideout in 2011 – swam ashore from speedboats before members of the Islamist militant group al-Shabaab rose for dawn prayers, officials and witnesses said. They stormed a two-storey beachside house in Barawe said to be occupied by foreign members of al-Shabaab and battled their way inside, a fighter who gave his name as Abu Mohamed told AP. There was a heavy gunbattle at about 2.30am on Saturday, according to locals in Barawe, about 60 miles south of the capital Mogadishu. Mohamed Hassan, a schoolteacher, said: "Nearly an hour before the morning prayer I heard dogs bark and I got up, but within minutes I heard small gun fire towards the direction of the beach. I raised my ears up as the shooting continued and continued. Soon it became like an exchange of fire. Then I heard one big explosion and two other explosions occurred. I could not go outside so I remained in my room to wait what was happening."

Hassan said the shooting he could hear was that of al-Shaabab's fighters because he understood the US forces were using silencer guns so no one could hear their shooting. "In the morning, we saw people gathering near the house the US forces targeted and there was a lot of blood everywhere. The al-Shabaab fighters told us not to go to the direction of the house. I saw one dead and two others injured but they were not very critical."

No one in Barawe town could have imagined such an attack, he added, and they kept saying only "white soldiers attacking Barawe town". Local residents said late on Saturday that al-Shabaab deployed additional fighters in Barawe to keep guard at the beach where the navy Seals landed.

US officials told AP that the Seals encountered fiercer resistance than expected, so after a 15- to 20-minute firefight, the unit leader decided to abort the mission and they swam away.

A local resident, Haji Nur, said he saw military equipment which al-Shabaab claimed to have confiscated from the soldiers. "I saw in the centre of the town a crowd of people gathering and looked at three rounds of M16 ammunition, one US-made hand grenade and one also a bulletproof jacket."

Al-Shabaab, which has a formal alliance with al-Qaida and claimed responsibility for the Nairobi mall killings that killed at least 67 people, posted what it claimed were pictures of the equipment on the web.

Sheikh Abdiasis Abu Mus'ab, a spokesman for al-Shabaab, said: "Early on Saturday morning, around 2am, white soldiers attacked a house resided in by some members of the Mujahideen leaders in Somalia. They came from a waiting speedboat from warship and as they were approaching the house, our Mujahideen fighters repulsed them. They ran away. We chased them until they have reached the seaside where they urgently boarded their speedboats."

Mus'ab said one al-Shabaab member had died and claimed that the Seals lost a "senior officer". US officials said there were no US casualties in either the Somali or Libyan operation.

A resident of Barawe who gave his name as Mohamed Bile told the AP that militants closed down the town in the hours after the assault, and that all traffic and movements have been restricted. Militants were carrying out house-to-house searches, likely to find evidence that a spy had given intelligence to a foreign power used to launch the attack, he said.

A Somali intelligence official was quoted as saying that Godane, the al-Shabaab leader also known Mukhtar Abu Zubeyr, was the target of Saturday's raid. Mohamed Ansari, a former al-Shabaab member now working with Somalia's counter-terrorism unit in Mogadishu, said: "Godane is the only big fish in Barawe to hunt. Godane as the top leader of al-Shabaab and the only planner of the group's operations is seen as the mastermind of Westgate mall siege in Nairobi."

Unlike his Libyan counterpart, Somali prime minister Abdi Farah Shirdon welcomed the US intervention. "We have close cooperation with the world, especially the western countries in the fight against al-Shabaab," he said in Mogadishu on Sunday. "We welcome any operation to hunt the terrorist leaders and we are at the forefront. Al-Shabaab is a Somali problem, a regional problem and world problem."

The dual raids were a vivid of expression of how the US has quietly been building its military capacity in Africa. Kerry, who is in Indonesia for an economic summit, said: "We hope that this makes clear that the United States of America will never stop in the effort to hold those accountable who conduct acts of terror. Members of al-Qaida and other terrorist organisations literally can run but they can't hide."

But a diplomatic source focused on Somalia said: "This is knee-jerk stuff and smacks of a massive failure of intelligence. Are extrajudicial killings and covert kidnapping raids the best way of dealing with the problem? Why is the international response so feeble?"

But Dr Adekeye Adebajo, executive director of the Centre for Conflict Resolution in South Africa, said that while it was in the interest of African governments to fight terrorism, he does not "think the heavy-handed and unilateral way the US acts is helpful and it risks causing further instability, especially where there are weak governments like in Libya and Somalia".