Islamic State (IS) fighters have been targeted by other radical Islamist groups in Libya who have now declared a jihad against the extremist group.

Militants loyal to IS have gained ground in Libya, taking control of the central city of Sirte, but in their eastern stronghold of Derna they have run foul of other radical Islamist groups.

IS fighters have been accused of killing a leader in an umbrella group of Islamist militias, Majlis al-Shura, because he apparently refused to swear allegiance to the supreme leader of IS in Iraq, Abu Bakr al Baghdadi.

After the infighting that followed, locals said at least 20 IS fighters were killed and Majlis al-Shura declared a violent jihad, or holy war, against the IS men.

"We, the Mujahideen of the Derna council, have warned those Islamic State militants to stop their criminal acts but they were committing even more crimes," Majlis al-Shura said in a statement.

Angry supporters of the Majlis al-Shura stormed several buildings held by IS during the fighting, while shops closed and frightened residents stayed indoors on Wednesday, according to residents.

The battle appeared to have been the largest yet between IS and rivals in Derna.

Forces loyal to Libya's official government later sent an air strike against suspected IS positions in Derna, said Nasser al-Hassi, spokesman for Benghazi's air base.

IS and other Islamist groups have thrived in Libya since two governments began fighting for control of the oil producer, leaving a security vacuum four years after the ousting of Muammar Gaddafi.

The rapid rise in Libya of IS, which first emerged in Syria and Iraq, is worrying Western powers who fear the militant group will forge a base just across the Mediterranean from mainland Europe.

IS has expanded its territory to include the whole of the central city of Sirte, and Harwa to the east.

The group has carried out attacks on oilfields and embassies and also claimed the killing of dozens of Egyptian and Ethiopian Christians.

ABC/Reuters