Deadlines for a retreat of Houthi troops in Yemen, agreed in talks last month, have had to be delayed, the UN special envoy for Yemen, Martin Griffiths, has said. He also conceded plans for prisoner exchanges have not gone to plan.

Griffiths also had to deny that the retired general Patrick Cammaert, appointed by the UN to implement the ceasefire in the Red Sea port of Hodeidah, had quit due to disagreements with Griffiths’s team.

Griffiths confirmed Cammaert, a retired Dutch general, was leaving after only weeks in the job, but said he had always been on a short-term contract and there had been no dispute between the two men after the general’s convoy was fired upon in Hodeidah.

Cammaert had been struggling to persuade the two sides to attend a regional coordination committee, seen as central to building trust, and ironing out disagreements over what was agreed in peace talks in Stockholm last month.

Griffiths also rejected a call from the Saudi-led coalition fighting the Houthis for the UN to declare the ceasefire is over. Such an announcement would be taken as a signal for the Saudi-backed forces to renew their offensive to capture the Houthi-held port city, seen by the Saudis as central to the Houthis’ effort to remain in power in the populated north of the country.

For nearly four years, the Saudis and the United Arab Emirates have been giving financial and military backing to the UN-recognised government of Abd Rabbu Mansour Hadi to regain power but have found the Iranian-backed Houthi resistance stronger than anticipated.

Speaking to the Arab newspaper Asharq al-Awsat, Griffiths said “timelines have been extended. The original deadlines were rather ambitious. We are dealing with a complex situation on the ground.” But he said he “categorically rejected” calls to name the party obstructing the Stockholm agreement.

“I believe that the political leadership of both parties is determined to put an end to the suffering of the Yemeni people.” He accepted “the situation was volatile”, but said “the ceasefire is generally holding despite any security incidents that have been taking place.

“General Cammaert’s plan was to stay in Yemen for a rather short period of time to ... lay the ground for establishing the Hodeidah mission. All the speculations about other reasons for the general’s departure are not accurate.”

The Yemeni foreign ministry said in a statement on Saturday that the Houthis had dodged the implementation of the Hodeidah agreement, refused to open safe corridors for the delivery of aid, and bombed Red Sea wheat silos, destroying badly needed aid.

Yemeni government ministers personally met the UN secretary general, António Guterres, on Friday to demand he do more to enforce the agreement.

Griffiths’s gloss on the state of the talks either reflects subtle progress towards a ceasefire only visible to those deeply involved in the talks, or the optimism required for a diplomat desperate to preserve the best and only chance for peace in Yemen. He is travelling to Yemen again this week to push for progress.

In London 14 British-based aid groups urged both sides in the civil war to remember that the people of Yemen come first, adding commercial aid is not yet flowing to the level needed to restore vital imports of food and medicine. A large number of ships are waiting outside the port seeking permission to dock.

Speaking from Hodeidah, Salem Baobaid, Islamic Relief’s head of office coordinator, said: “Until now, little has changed for ordinary people. After the months and months of bombing, shelling and starvation, it will take much more than a ceasefire to start breathing life into people who have been living on the edge of death for so long.

“Things are so bad that large groups of people have started living in squalid, toxic conditions on the edge of the city’s main, highly contaminated garbage dump – just so they can forage for scraps.”

He said members of his aid team had been killed by stray bullets and shelling was continuing.

Baobaid said he had recently met Faiza, an 11-month-old girl weighing 5kg who had to come into the centre with her mother. “She barely moved, and had wide opened eyes … She barely cried. Her face was so pale it scared me. She had very bad diarrhoea and her mother was so exhausted that she could not breastfeed her properly.

“When I first saw her, I thought she only had a few days left in this world. Luckily, we were able to help her and she is now out of danger.” He said there were 400,000 other children in the same state.

In common with other aid workers, he said difficulties in getting permits to transport aid remained a huge problem with roadblocks every 20 minutes or so on the road between Hodeidah and Yemen’s capital, Sana’a.

Other aid workers said the conflict was shifting from Hodeida and areas previously outside the fighting were now being sucked into the battles.