Top former civil servants have warned Theresa May that squabbling cabinet ministers, unrealistic expectations and an overburdened administration risk derailing her hopes of a smooth Brexit.

May faces renewed calls to compromise or face the prospect of crashing out of the European Union. On the eve of the next round of Brexit talks, the prime minister is under pressure to show she is willing to alter key aspects of her EU exit plans, which no longer command majority support in parliament.

In a warning over the scale of the challenge now facing the government, Lord O’Donnell, the former cabinet secretary, writes in the Observer that Britain is in for a “rough ride” unless cabinet ministers unite and back a long transition deal to soften the impact of Brexit.

“The EU has clear negotiating guidelines, while it appears that cabinet members haven’t yet finished negotiating with each other, never mind the EU,” the crossbench peer warns. He calls on ministers to “start being honest about the complexity of the challenge”.

“There is no chance all the details will be hammered out in 20 months,” he warns. “We will need a long transition phase and the time needed does not diminish by pretending that this phase is just about ‘implementing’ agreed policies as they will not all be agreed.”

Sir Nigel Sheinwald, Britain’s former ambassador to the EU, told the Observer that there was now a “one-in-three” chance of Brexit talks collapsing unless the UK drastically reset its plans. He warned that the government still appeared to be “defending a set of propositions that feel tired and totally unrealistic and removed from the realities of these negotiations”. He said a serious offer on future payments was needed, as well as support for a “very substantial transition period” that was similar to Britain’s current EU links.

“I think [the negotiations] could break down quite quickly if, after the autumn, our position on these fundamental issues – the financial settlement, and then the fundamentals of our future relationship – don’t move more in the direction of common sense and economic sense,” he said.

“I don’t think it’s the most likely option, but I think there’s a one-in-three chance at least that that’s the way it will go. And I would say, where we are today, it feels to me that it is at least something like that. And that’s a very sizeable risk.”

Tony Blair also intervened on Saturday to suggest that EU leaders may be prepared to alter the bloc’s free movement rules to keep Britain inside the single market. He said senior EU figures had indicated they could be flexible, but that Britain had to be willing to compromise.

A spokeswoman for the Department for Exiting the European Union (DExEU) defended the government’s plans. “Our aim is to build a deep and special partnership with the European Union, as the EU’s closest friend and neighbour,” she said. “As we negotiate to leave the EU, we are seeking the best possible deal for the UK, one that gives citizens and businesses as much certainty as possible.”

The interventions come after a week in which Amyas Morse, head of Britain’s public spending watchdog, warned a lack of leadership meant the government’s response to Brexit risked falling apart “like a chocolate orange”.

While May has survived the immediate threat to her premiership in the wake of the disastrous election result, Whitehall insiders and some ministers believe she is entering a crucial period between now and the autumn in which she will have to show some willingness to modify her Brexit plans.

An Opinium poll for the Observer found that most voters (57%) believe May should resign before the next general election or earlier. Her net approval ratings remain dire, with 51% disapproving of the way she is handling her job and only 30% approving. Overall, Labour has a lead of two points over the Tories.

O’Donnell, who served as Britain’s senior civil servant under Tony Blair, Gordon Brown and David Cameron, said that Whitehall needed a boost in resources to meet the challenge ahead. “The Treasury has allocated over £400m for extra staff and knows that much more will be needed,” he writes.

Despite reassurances from the government, there are still concerns in the scientific community over the plan to leave Euratom. Writing in the Observer on Sunday, Ian Chapman, chief executive of the UK Atomic Energy Authority, warns the move must not hit Britain’s reputation as a world leader in nuclear science.

“It is imperative that we rapidly find a solution for a continued relationship with Euratom,” he writes. “One of the options available is an association agreement with the organisation – a bespoke arrangement where the UK could remain inside Euratom with certain conditions and constraints.”

Meanwhile David Isaac, chair of the Equality and Human Rights Commission, suggested that there would be a surge in cases brought under the Human Rights Act, which guarantees a person’s entitlement to privacy and a family life. While the government has guaranteed many of the rights for EU citizens living in the UK post-Brexit, some could be lost, including being permitted to bring in family members, such as spouses or elderly relatives.

“[This] should not be used as a bargaining chip in these negotiations,” Isaac said. “The government must respect the human rights of EU nationals – not just because they could face a mountain of legal challenges if they don’t, but because it’s the correct thing to do.”

Martha Spurrier, director of the human rights group Liberty, said: “There will be interesting legal arguments about that and whether it’s legally fair to say to someone, ‘well the day before Brexit you had these rights and now an executive decision has been taken and the day after it you no longer have [them], which means you might be deported’. There will be EU citizens who find themselves in immigration detention, facing the same brutal enforcement mechanisms that the government inflicts on other migrants.”

A Home Office spokesman said the department had “made it explicitly clear that no EU citizen currently in the UK lawfully will be asked to leave at the point we leave the EU”.

“In fact, we will allow up to two years for people to regularise their status,” the spokesman said. “We look forward to discussing this in the next round of negotiations as we pursue an agreement on this important issue as soon as possible.”