The United Nations special envoy for Yemen has urged the country's warring sides to withdraw their troops from the port city of Hodeidah, a lifeline for millions of starving Yemenis, after aid agencies warned conditions in the impoverished country were deteriorating fast.

Martin Griffiths, who arrived in the rebel-held capital, Sanaa, on Monday for his third trip to Yemen this month, said while there had been "changes in timelines" in both the proposed troop pullout and a prisoner swap, the "momentum is still there".

Yemen's warring sides had agreed to a series of confidence-building measures last month in Sweden that included a plan for the Houthis to withdraw from the contested port city of Hodeidah and place it under the control of an interim entity.

The two sides had also exchanged the names of some 16,000 prisoners of war, and representatives from both the Yemeni government and Houthis rebels had said they expected the transfers to commence by January 19.

But little further progress has been made, risking the unravelling of the peace efforts.

"In light of the facts that the timelines were rather ambitious and we are dealing with a complex situation on the ground," Griffiths told Saudi-owned newspaper Asharq Al-Awsat.

"That momentum is still there, even if we have seen the timelines for implementation extended, both in Hodeidah and with regard to the prisoner exchange agreement."

"More than any time in the past, there is a political will demonstrated by all parties to put an end to this conflict," he said.

"What we need to see now is the implementation of the provisions of the agreement, fully and rapidly."

'Prison without walls'

His comments came shortly after aid agencies, meeting in London, said Yemenis were still struggling to feed their children in what has become the world's worst humanitarian crisis.

"I think we need to be very clear that we need a political solution to this conflict," Isabelle Moussard Carlsen of Action Against Hunger said.

At the meeting in London, 14 aid agencies called for urgent action on the humanitarian crisis.

"It is what I like to call a prison without walls for the people living in the country at the moment. It is a difficult situation where people are struggling to buy their daily rations to be able to feed their children," Yemen's Oxfam campaign manager Awssan Kamal told Reuters.

Kimberley Brown of the British Red Cross said 85,000 children had lost their lives and malnutrition was taking a huge toll.

"I know from my colleagues that the situation is absolutely deteriorating at the moment," she said.

The war has been at a stalemate for years, with the Saudi-UAE coalition and Yemeni forces unable to dislodge the Houthi movement that controls the capital city and most urban centres.

The coalition has twice attempted to capture Hodeidah port since last year to force the Houthis to negotiate but held off from a full-blown assault amid fears that a disruption to supply lines would trigger mass starvation.

The coalition entered Yemen's war in March 2015 in an attempt to roll back advances made by the Houthis, a group of Shia rebels who overran much of the country in 2014.

With logistical support from the United States, the coalition has carried out more than 18,000 raids on Houthi-held areas in an attempt to reverse their gains.