Libya's former interim prime minister Mahmoud Jibril has won a landslide victory in the country's first democratic election, early figures show, defying expectations that the Muslim Brotherhood would sweep to power.

Jibril, a moderate who led Libya's National Transitional Council (NTC) last year, won a clear majority of votes across most of the country, the first on Monday suggested.

Giving details in three constituencies, the country's election commission said Jibril received 6,798 votes in Janzour, a suburb in western Tripoli. The Muslim Brotherhood came second with 2,423. In the former pro-Gaddafi town of Zlitan, west of Misrata, he got 19,220 votes with the brotherhood receiving 5,626.

Unconfirmed reports suggest Jibril won around 80% across Tripoli, with strong results in the south and 60% in Benghazi, the eastern city where Libya's revolution began. Voters in Misrata, however, elected their own local candidate and his Union for the Homeland party.

Jibril's victory bucks the trend for post-Arab spring elections, which have seen Islamist parties win power in neighbouring Egypt and Tunisia. But in Libya the two main Islamist parties – the Justice and Development party and Al-Watan, led by the former rebel militia leader Abdel Hakim Belhadj – did less well than predicted.

Speaking on Sunday night at his party's headquarters in a villa in western Tripoli, Jibril said he was willing to work with all of the country's diverse political forces, opening the door to a grand coalition. His officials privately confirmed that their own polling indicated Jibril's National Forces Alliance had taken a commanding lead in Tripoli.

"We extend an invitation, continued as before, to other political forces to come together in one coalition under one banner. This is a sincere call for all political parties to come together," he said.

Holding out an olive branch to Islamic parties, he added: "There are no extremists." Jibril also mentioned the minority Toubou and Amazigh (Berber) populations, who have complained of discrimination in the past.

Since the overthrow of Gaddafi last year, ethnic conflicts have raged in several parts of the country. Jibril said: "They have demonstrated their patriotism by voting."

Speaking in Arabic and English, Jibril rejected claims from some conservative clerics that his party was secular: "Some media channels started referring to the National Forces Alliance as liberals – that's not true. It is composed of different political formations."

At least one party, the Misrata-based Union for Homeland, said it would not work with a government that chose Jibril as a prime minister. Abdurrahman Sewehli, leader of the Union for Homeland party which won the most votes in Misrata, Libya's third largest city, beating the Muslim Brotherhood's party into second place, said Jibril's former ties to Gaddafi could not be overlooked.

"Misrata is the only place in Libya where Jibril is getting nothing," he said, after Jibril's allliance came fourth.

"Symbolism is very important. Mr Jibril is still representing the old regime." Jibril worked as an economics adviser to Gaddafi's son Saif al-Islam, resigning at the start of the revolution.

Final results are not expected until later this week. One of Jibril's urgent tasks will be to assuage federalist supporters in the east of the country, who ransacked several polling stations on Saturday, killing two and setting light to ballot papers. The federalists want a greater share of seats in the national congress.

"There should be a serious dialogue [with the east]. As there is a sincere wish on their part and on our part I think we can reach a compromise," Jibril said on Sunday, declining to specify what role he saw for himself in Libyan politics. The new congress's chief task will be to appoint – or hold elections for – a working group to write a constitution, with new elections due to be held in 2013.

Further violence was reported on Monday in several areas of Libya.

There were claims that the Sufi shrine of Zuhayr Ibn Qais al-Balawi, said to have been a companion of the prophet Muhammad, was destroyed by Salafists in the eastern town of Derna. The town has a reputation as an Islamist stronghold.

After two Misratan journalists were detained in the formerly pro-Gaddafi town of Bani Walid, Misratan armoured columns were deployed on Monday near the town demanding their release. And street battles broke out in the southern town of Sabha with one man reported dead. The fighting is evidence that Libya remains a fractious country and that security will remain Jibril's key challenge.