Greece has ordered the expulsion of the Libyan ambassador in an escalation of a dispute over a controversial deal signed between Libya’s UN-supported government and Turkey on maritime boundaries in the Mediterranean.

The Greek foreign minister, Nikos Dendias, said the ambassador, Mohamed Younis AB Menfi, had been summoned to the ministry in the morning to be informed of the decision, and was given 72 hours to leave the country. Turkey’s foreign minister condemned the move as outrageous.

Rare direct talks between the Turkish president, Recep Tayyip Erdoğan, and the Greek prime minister, Kyriakos Mitsotakis, on the margins of the Nato summit on Thursday failed to resolve the row over the sudden Turkish-Libyan claim to an exclusive economic zone.

Under the terms of a deal reached between Libya and Turkey on 27 November, Turkey gets access to a zone across the Mediterranean, ignoring the objections of Greece, Cyprus and Egypt, which lie between Turkey and Libya geographically.

Mitsotakis said the claim had no legal basis and insisted it would collapse.

He told the Greek parliament: “They are oblivious to history and geography as they do not take Greek islands into account,” adding that Ankara’s move had pushed the Nato country into “unprecedented diplomatic isolation”.

The Libyan foreign minister said Greece’s diplomatic move was unacceptable.

The row is part of a long-running dispute between Turkey and Greece over drilling rights in the Mediterranean, as well as a byproduct of the need of the Libyan UN-recognised government in Tripoli to remain close to Turkey as one of the few outside actors backing its effort to hold on to the capital.

Greece and Turkey are at odds over a host of issues ranging from mineral rights in the Aegean Sea to the future of a reunified Cyprus.

Tensions are running high because of Turkish drilling off Cyprus, and the EU has prepared sanctions against Turkey in response.

Cyprus has said it will take Turkey to the international court of justice in the Hague, and has enlisted the support of the EU. But Greece has said it will not tolerate any drilling south-east of Crete, and has warned it will send its naval forces to prevent such a move if necessary.

After meetings earlier this week with the new EU external affairs chief, Josep Borrell, the Greek foreign minister said: “Turkey, through its violations of international legality, is constantly creating disputes with all of its neighbours. There is not a single neighbour of Turkey that does not have disputes with it. I think this is a little too much, and is telling.”

Turkey claims the UN convention on the law of the sea allows a country to stretch its territorial waters by 12 nautical miles out to sea, but that when it comes to an exclusive economic zone, where it has the rights to fishing, mining and drilling, the area can extend for an additional 200 miles. However, if the maritime distance between the two countries is less than 424 miles, a bilateral deal, such as the one it has struck with Libya, is needed to determine a mutually agreed-upon dividing line for their respective exclusive economic zones.

The Turkish-Libyan deal is largely designed as a legal defence against Cypriot and Greek claims to the waters. Libya is diplomatically dependent on Turkey, but the Tripoli government is taking a risk if it alienates key partners in the EU.

Turkey has been supplying arms to the Tripoli government of national accord (GNA), just as the United Arab Emirates and Egypt have been providing direct military support to the warlord Field Marshal Khalifa Haftar, who is mounting an increasingly effective effort to dislodge the GNA. A UN report due on Monday will say a country is responsible for the bombing of Tripoli, but will not name the UAE.