Months of futile diplomatic tussling, UN deadlock and finger-pointing over Syria have boiled down to a dramatic, last-ditch effort this weekend to cut a deal between the US and Russia that eases President Bashar al-Assad from office and replaces him with an inclusive, transitional government that can halt the spiral towards all-out civil war.

Barack Obama's administration first floated the idea of ditching Assad while simultaneously guaranteeing Russia's interests in Syria more than a month ago. Despite Moscow's rebuffs, the White House has kept at it. Obama spent two hours discussing Syria with a sceptical Vladimir Putin, Russia's president, at this month's G20 summit in Mexico.

US officials did not pretend Putin was won over. But they did claim headway in identifying areas where US and Russian interests coincide, principally preventing a chaotic implosion and a regional war. "We agreed that we need to see a cessation of violence, that a political process has to be created to prevent civil war," Obama said. "We have found many common points on this issue," said Putin.

After follow-up meetings, Hillary Clinton, US secretary of state, and Sergei Lavrov, Putin's foreign minister, have agreed to attend an international summit on expediting Syria's political transition to be convened in Geneva on Saturday by the UN envoy, Kofi Annan. Clinton and Lavrov will meet privately beforehand in St Petersburg on Friday.

US-Russian agreement on the way forward in Syria is crucial. Russia is the Syrian regime's most powerful ally and protector, its main arms supplier, and a veto-wielding member of the UN security council. Its influence within the regime is unmatched. For its part, the US is the world's foremost military and economic power with extensive Middle East interests, including guaranteeing Israel's security and safeguarding its energy supply.

For the past 18 months they have been at loggerheads. Now, despite hardline statements ahead of the meeting, they appear to be trying to work together.

On the American side, the need for a deal is more pressing. Obama, facing a tough re-election battle this autumn and with his domestic record assailed from all sides, could use a big international win. His handling of the Arab spring was not impressive. Last week, as rightwingers see it, he "lost" Egypt, Washington's main Arab ally, to the Muslim Brotherhood. Mitt Romney, his Republican challenger, is describing Russia as America's biggest strategic threat. Thus the meaningfulness of Obama's attempt to reset relations with Moscow is also in the balance.

A transition deal in Syria would suit Obama for a host of other reasons. It would defuse criticism from American interventionists about US inaction. It would also help secure thestabilityof Iraq, on which so much American blood and treasure was spent in the past decade. It would prevent the spooked, volatile leaders of Turkey, a valued Nato ally, sliding into some kind of regional conflict.

Most of all, by stabilising Syria under a potentially more amenable regime with less allegiance to Iran, Obama might hope to lessen pressure from and on Israel to attack Tehran's nuclear facilities this autumn. "It is the strategic relationship between the Islamic Republic and the Assad regime [not Iran's unknown nuclear capabilities] that makes it possible for Iran to undermine Israel's security," said former US diplomat James Rubin. "The collapse of the Assad regime would sunder this dangerous alliance."

For Russia, a Syria deal with the US has attractions, including Putin's visceral, often unthinking distrust of all things American. Russia's interest dictates a settlement in Syria that sustains its influence on its Arab ally, keeps its Mediterranean port facilities open, and maintains its business and arms sales links. As Assad loses ground to the opposition, these interests appear threatened.

Russia's broader interest (and national pride) demands that it prevent another Libyan or Iraq-style western military intervention, that it assert its authority as a major world player, and that it now move to limit the damage done to its links with the Gulf states and others by its apparent support for Assad's bloody repression. Some kind of national unity deal, under UN auspices, that could be passed off as a Syrian, not an American, solution would serve all these desired ends.

"Russia's leaders have said repeatedly that their goal is to guard against instability, not to support Mr Assad," said Ellen Barry, reporting from Moscow. "They have signalled that Russia would accept a change of leadership in Syria, but only if devised by Syrians and not imposed from outside."

Edward Burke of the Centre for European Reform makes a similar point: "Moscow knows the writing is on the wall for the Assad regime and that its slow demise will likely precipitate an increasingly deadly civil war that will damage Russian interests … An attempt by Russia to negotiate a political transition should be welcomed. Moscow has made it clear that it future relations are not tied to the power status quo in Damascus."

The scene is set for a Clinton-Lavrov meeting that could still ring the death knell for the Assad regime. Maybe they can pull it off; maybe they cannot. Whatever the outcome, democrats will note that Obama and Putin have something else in common: a deep wariness of the politically unquantifiable Syrian opposition and a pragmatic disinterest in the wishes of the Syrian people.

This backroom deal, if it happens, has little to do with building a democratic Syria. It has everything to do with fixing a problem that is upsetting the world order as decreed from Washington and Moscow.