Tension between Turkey and Syria is worsening as thousands of refugees from repression by president Bashar al-Assad flee across the border

Officials in Ankara were watching closely as Syrian forces deployed in a village close to the border, Khirbet al-Jouz, after Turkey had flatly rejected an appeal from Damascus to moderate its increasingly angry public comments about the crisis.

Recep Tayyip Erdogan, Turkey's prime minister, has attacked the repression as "savagery" and urged Assad to sack its military mastermind, his brother Maher, and implement genuine reforms in the spirit of the "Arab spring".

But Erdogan has so far failed to demand that the Syrian president stand down – as he did with Egypt's Hosni Mubarak and Libya's Muammar Gaddafi.

Still, officials, diplomats and analysts say that a bilateral relationship that has flourished politically and economically in recent years is now badly, perhaps irreparably, damaged.

"The rapprochement between Erdogan and Assad has pretty much broken down," said Fadi Hakura of the Chatham House thinktank in London. "Turkey is becoming ever more strident and direct, and this is causing deep unease in Damascus."

On Wednesday the Syrian foreign minister, Walid al-Moallem, publicly urged Turkey to reconsider its hostile stand, but the Turkish ambassador immediately dismissed the call.

"The relationship has become very frosty," said Hugh Pope, Istanbul director for the International Crisis Group. Erdogan had been urging Assad to make domestic changes since before the uprising began in March.

Ahead of Assad's speech on Monday, Ersat Hurmuzlu, an adviser to president Abdullah Gul, said Assad had a week in which to act – but Turkish officials were left disappointed by Assad's lacklustre performance.

"We had high expectations that the Syrian president would deliver," said a senior Turkish official. "But we were disappointed."

The Turkish-Syrian honeymoon began when Erdogan came to power in 2003, and cooled Turkey's once close relations with Israel while making overtures to the Palestinian Islamist movement Hamas.

Following his re-election this month he vowed to reach out to the Middle East and beyond to promote "justice, the rule of law ... freedom and democracy", distancing himself from the traditional stabile friendships with Arab dictators.

"When Turkey has to make a choice between regimes and people," the senior offiical said, "it will always be on the side of the people."

British officials describe a "meeting of minds" when David Cameron spoke to Erdogan last week. The US and Britain say that they hope a policy rethink in Ankara will also include a distancing from Iran and its alleged nuclear ambitions.

"The Turks are increasingly unhappy with what is happening in Syria," said a western diplomat. Another consequence has been a renewed warming of relations with Israel after the row over the Gaza aid flotilla last year, when a Turkish ship was boarded on the open seas by Israeli commandos and nine activists killed.

Syria was furious last month when Turkey hosted a high-profile conference of Syrian opposition activists in Antalya.

Turkish officials deny any plan to create a "security zone" on the border – a sensitive step given memories of Ottoman days (and the Turkish border province of Hatay, which Syria continues to claim as unjustly ceded in a plebiscite), and especially without an international mandate.

Turks recognise the change that has taken place. "Turkey's close rapport with the US regarding ... Syrian politics shows Turkey has completely parted company with Assad," commented Nihat Ali Özcan in the Hurriyet daily.

"Erdogan doesn't want another diplomatic crisis in the context of Syria, like the one instigated by the nuclear issue with Iran. We can say that he is ideologically much closer to the Muslim Brotherhood than Assad."

The US has praised Turkey for its "big heart" in helping refugees. "But clearly, Turkish patience appears to be wearing thin, and we share all of their humanitarian and political concerns," said a US state department spokesman.

"Erdogan is in a very challenging position," Hakura added. "He is trying to react to facts on the ground in Syria, but at the same time he hasn't called on Assad to step down. The more violence escalates, the more difficult his position will be."