The United Nations says Libya has apologised for the detention of one of its officers on suspicion of "black magic".

Guards at a Tripoli prison arrested Ahmed Ghanem, an official observing the trial of two sons of Muammar Gaddafi, earlier this month, saying they suspected him of having written materials containing spells or enchantments.

The UN stability mission in Libya (Unsmil) said it has had an apology and assurances from the government that it respects an agreement giving UN staff immunity from arrest.

"Unsmil was dismayed by the detention of Mr Ghanem for an hour and a half as well as what followed in terms of defamation and false accusations through some social media pages," said a UN statement. "The mission affirms that it scrupulously respects the Libyan authorities' legal, security and administrative procedures when its representatives attend trials."

The official was arrested at the trial of Saif al-Islam and Saadi Gaddafi, who are accused along with more than 30 former regime officials of war crimes. He was later released after guards decided their suspicions were unfounded. The UN said it has had messages of support from diplomats and members of Libya's judiciary.

"The mission described the accusation of sorcery not only as groundless, but also as ridiculous," said a UN spokesman.

On Tuesday, the international criminal court denied Libya's appeal for the right to try Saif, saying Tripoli lacked conditions to ensure a fair trial, and ordered the government to send him to The Hague.