The foreign ministers of Saudi Arabia and Iran have shaken hands at a meeting of the main grouping of the world's Muslim nations despite the rival nations being locked in a bitter feud, the ISNA news agency has reported citing state television.

Saudi Foreign Minister Adel al-Jubeir and Iranian counterpart Mohammad Javad Zarif greeted each other on Tuesday at an extraordinary meeting in Istanbul of the Organisation of Islamic Cooperation (OIC) on recent tensions in the occupied city of Jerusalem.

The two ministers held brief talks on the sidelines of the meeting with Zarif explaining that the move was within "diplomatic conventions" and a sign of his "mutual respect and long-standing friendship with Jubeir", the news agency reported.

The Sunni-ruled kingdom and Shia-ruled Iran have long competed for influence in the Middle East. The two rivals have backed opposing sides in Syria and Yemen, where ongoing civil wars have claimed thousands of lives.

On June 5, Saudi Arabia, Bahrain, the United Arab Emirates and Egypt cut ties with the Gulf state of Qatar accusing it of fostering ties with Iran and backing extremism - Doha has categorically denied the allegations.

The OIC on Tuesday accused Israel of staging provocative actions and inflaming tensions with the Palestinians in a crisis over security measures at a key holy site in Jerusalem.

Israel angered the Islamic world by installing metal detectors and security cameras at the Haram al-Sharif holy site in Jerusalem, known to Jews as the Temple Mount, following a July 14 attack in which gunmen killed two policemen.

The move sparked Muslim protests and deadly unrest, and last week the Israeli government removed the detectors and cameras. The site includes the al-Aqsa Mosque, one of the holiest sites in Islam.

A joint communique issued after the gathering said the meeting "strongly condemns Israel's recent provocative actions" at the holy site.

It accused the Jewish state of "employing collective punishment measures and the use of lethal and excessive force against peaceful Palestinian worshippers" at the site.

And despite the Israeli climbdown on the metal detectors, the OIC statement accused the government of "persistent public statements... that incite religious sensitivities, feed tension and incite violence".

It urged world powers not to support or encourage Israel's "illegal colonisation and annexation" of East Jerusalem, which Israel occupied in 1967 before later annexing.

Palestinian foreign minister Riyad al-Malki told the meeting that Israel's move to remove the detectors was a "small victory in the long battle for freedom".

But he accused Israeli Prime Minister Benjamin Netanyahu of seeking to change the longstanding agreement whereby only Muslims are allowed to pray inside the Al-Aqsa mosque compound, although anyone can visit, including Jews.

"Netanyahu will try again to impose his status quo and we should prepare for the next round which could come very soon and be very nasty," he added.

Turkey has full diplomatic relations with Israel after resolving last year a crisis in ties but President Recep Tayyip Erdogan remains vehemently critical of the Jewish state's policy towards the Palestinians.

Last year Turkey and Israel ended a rift triggered by Israel's deadly storming in 2010 of a Gaza-bound ship that left 10 Turkish activists dead. The two sides have since embarked on a close energy cooperation venture to pipe Israeli gas to Turkey.