The Yemeni regime's indiscriminate machine-gunning of demonstrators in the capital, Sana'a, and the opposition's furious reaction, suggests the country's eight-month-old crisis may be coming to a head. But the interests of two key outside players, the US and Saudi Arabia, remain focused more on strategic security and terrorism concerns than on spreading democracy and prosperity in the Arabian peninsula.

The US stepped up pressure last week for an end to the rule of President Ali Abdullah Saleh, urging the regime to accept a previously formulated political transition deal within seven days. The plan, mediated by the Saudis and other members of the Gulf Co-operation Council, calls for a government of national unity, presidential elections and a new constitution.

But Saleh, holed up in Saudi Arabia after an assassination attempt in June, has so far resisted the plan's key provision – that he step down and hand over power to his vice-president in exchange for immunity. Although the US state department said it was "encouraged" that Saleh had ceded negotiating authority to his deputy, there is as yet no sign that either the US or the Saudis are ready or willing to force his departure from the scene. If they were, they surely would have done it months ago, one way or another. This hesitation to definitively pull the plug, despite rising mayhem in Sana'a and other Yemeni cities, contrasts sharply with the way Washington ruthlessly cut Hosni Mubarak's legs from under him in Egypt. Indeed, Riyadh's unelected princelings strongly objected to Mubarak's treatment, viewing it as a dangerous precedent, and now appear doubly determined to prevent Saleh being disposed of in the same manner. Even if a transition deal is agreed, Saleh might remain in power almost indefinitely by finessing its terms.

Yet the principal reason why the regime is still in power is overriding US and Saudi worries about the potentially hugely destabilising ramifications of what may follow. The street-level, pro-democracy, Arab-spring struggle is but one aspect of a wider, more complex Yemeni conflict.

Others facets include power struggles between military and business elites, long-standing tribal rivalries, armed separatism in the south, Iranian-fomented Shia Muslim rebellion in the north, and most significant of all (for the Saudis and Americans), the tightening grip on Yemen of al-Qaida in the Arabian Peninsula – viewed by Washington as a bigger threat than al-Qaida in Afghanistan and Pakistan.

Saleh, for all his faults, was a staunch post-9/11 "war on terror" ally. The bottom-line fear shared in European capitals is that without Saleh Yemen could become a failed state, threatening Saudi Arabia's soft underbelly and thus western oil and gas supplies.

This may help explain why the US has been much more active bombing Yemen than reforming it in recent months. Obama administration officials told Karen DeYoung of the Washington Post this weekend that the US has "significantly increased" the number of Pakistan-style unmanned drone attacks on White House-approved al-Qaida targets, mostly in south Yemen. The CIA had been told to expand its Yemen operations and was building a new regional base, the officials said. Several drone attacks each week have been reported by local media. None of the attacks is publicly acknowledged.

In a speech at Harvard last week, John Brennan, Barack Obama's counter-terrorism adviser, indicated that Washington sees Yemen first and foremost as an important new battleground rather than a future bastion of Arab democracy. "The United States does not view our authority to use military force against al-Qaida as being restricted solely to 'hot' battlefields like Afghanistan," he said. "We reserve the right to take unilateral action if or when other governments are unwilling or unable to take the necessary actions themselves." Brennan maintained that this doctrine did not mean the US could use military force "whenever we want, wherever we want" – but it certainly points that way.

Amid the military escalation and political wrangling, another battle is taking shape in Yemen that could be more lethal than all that has gone before. According to a new report published by Oxfam, many Yemeni communities are "on the brink of disaster" due to rising hunger caused by rocketing food and fuel prices. Child malnutrition in Yemen, already the third highest in the world, is rising. Making matters worse, about 90,000 people have been displaced by fighting in the south.

But instead of increasing humanitarian relief and other assistance to Yemen as the crisis deepens, the World Bank has cut back on aid, citing the uncertain political and security situation. The UN and other agencies have also been handicapped by funding shortfalls as recession-hit wealthy countries keep their hands in their pockets. A UN-administered emergency relief fund only has 57% of its required funding for 2011.

With the political impasse continuing, and fighting flaring on all fronts, fears grow that Yemen may be reincarnated as Somalia II.