Talk about Israel to all but its most severe critics in America and whatever they say it's likely to be underpinned by an assumption that the US is the Jewish state's best friend. Doubts may emerge about Gaza and settlements but any criticism is invariably wrapped in the assumption that the US has Israel's interests, particularly its security, at heart. It's not surprising given that American politicians pledge their loyalty to the Jewish state over and over, and mean it.

They see things differently in Jerusalem. In a country permeated by fear and insecurity, Israelis define the rest of the world not by loyalties but by varying degrees of distrust. You can hear it among residents of Jewish settlements deep in the occupied territories and in the cafes of liberal Tel Aviv: angst over the perception of a new wave of antisemitism gripping Europe, the incomprehension over foreign condemnation of Israel's crimes in Gaza, the common agreement that the United Nations is a conspiracy against the Jewish state.

In all of this, the US emerges as the least distrusted country by far (Britain commands a much lower level of confidence). Israelis recognise that they have long counted on Washington to pay a good chunk of their military budget and provide diplomatic cover for the illegalities of occupation.But that is a far cry from trust, and what there was has been severely eroded since Barack Obama came to power as America catches up with the idea that Israeli government policies do not automatically equate with what is in Israel's interests or the promotion of peace.

That distinction has finally burst forth with the crisis over settlement building in East Jerusalem, exposing how far American political thinking has already shifted and the depth of Israel's prime minister Binyamin Netanyahu's failure to understand it. To the alarm of some Israeli leaders, what they regarded as a minor diplomatic blunder in announcing yet more ethnically exclusive housing in East Jerusalem has resulted not only in unusually forceful criticism from Washington, but the disturbing and unprecedented assertion by the White House that Israeli policies are jeopardising American interests and even endangering the lives of US soldier fighting in the Middle East. In days, the Israeli government has seen itself repositioned from valuable US ally in the war on terror to where it really belongs – as the primary obstacle to peace.

It was always striking to watch the smugness with which Israeli politicians sold Americans the idea that the only issue in considering the conflict with the Palestinians was what Ariel Sharon called, with deeply rolling Rs, the "tehrroar". The ever-expanding Jewish settlements, the annexing of land, expulsion of Palestinians, the killings of Arab children, the crimes of occupation, were discarded as an irrelevance. No one in George Bush's White House cared to challenge Sharon on that.

In Israel, ordinary people almost seemed programmed to repeat the wearying mantra that all the Jewish state ever wanted was peace but that all it ever got in return was blood and killing. That was a comfortable position for Israelis who, all too understandably at the height of the suicide bombings, found it difficult to reflect on why Palestinians might be blowing themselves up. But the myth of the relentless search for peace was important to the efforts of Sharon and Netanyahu to ensure that Americans did not call them on policies that deliberately or not – mostly deliberately – sabotaged its ever-diminishing prospects. Nowhere was this more clear than over the ever growing settlements.

Israel's apologists would have the world believe that the construction of homes solely for Jews in the occupied territories had no impact on the peace process and that the issue would be resolved in final status talks. But settlement construction is a litmus test of Israeli government intent because it is constantly changing the picture on the ground. The number of Jewish settlers in the West Bank has more doubled since the Oslo peace accords were signed in 1993. How is it possible to see that as anything but an attempt to preempt an agreement on Palestine's borders?

Watching Israel's barrier carve its way through Arab neighbourhoods of Jerusalem and the West Bank, its real intent was all too evident. The clues lay in the twists and turns that separated Palestinian villages from land then delivered up to Jewish settlers. As the barrier went up, there was barely a peep of protest from Washington.

But the ground has also shifted in America. The unconditional support for the Jewish state has been eroding in critical areas. Nowhere is that clearer than among American Jews, many of whom stayed quiet for so long out of loyalty to the country of Israel and Zionism, even as they grew increasingly disturbed at what was being done in their name by the militarists in Jerusalem.

In Washington, a group of Jews broke the taboo against criticising Israeli policy and launched J Street, an organisation that believes Israel's security lies in a just peace with the Palestinians. Founded by Jeremy Ben-Ami, a former domestic policy adviser to Bill Clinton with Israeli parents, it drew the backing of an array of community leaders, activists and academics. It reflects the doubts held by a significant proportion of American Jews about Jerusalem's militarist policies and, in particular, the deep-rooted obsession with settlements. Those doubts were heard and shared in the White House. The government in Jerusalem has been slow to grasp the significance of J Street because for so long many American Jews declined to voice their doubts out of loyalty. Buoyed by the certainty that it was irrelevant, Netanyahu dismissed the group as anti-Israeli and his foreign ministry snubbed a delegation of visiting members of the US congress sponsored by the Jewish organisation.

It was a mistake because it only confirmed in the minds of some in Congress and the administration that Israel will only move when pushed.

Recent events have confirmed that view as Netanyahu finally begins to understand the consequences of his mistake and climbs down, agreeing to a series of American demands aimed at kick-starting negotiations and putting the latest settlement plans on hold. It's a start, but Netanyahu is not a courageous leader nor an honest one. He pays lip service to a peace agreement but, like Sharon, spent the 1990s sabotaging attempts to reach agreements with the Palestinians and denouncing Israeli leaders who sought peace as traitors or worse. Netanyahu has still to commit himself to a genuinely independent Palestinian state. He will not do the right thing for the right reason.

Some in the US administration are straining at the leash to finally take him on and show him the way after continual humiliation of the US president. The question is whether Obama himself is now finally up for the fight.

Chris McGreal was the Guardian's Middle East correspondent for much of the last decade. He is now based in Washington.