When the EU’s suave chief Brexit negotiator Michel Barnier warned last week that he could hear the clock ticking down on Britain’s time to secure a comprehensive trade deal, his deliberately anxiety-inducing imagery played into a fear growing in the minds of many in Whitehall, the government and Brussels.

The concern gripping some involved in the saga of Britain’s extrication from the bloc is that a mixture of political indecision, a lack of leadership and a Whitehall machine struggling with the size of the task, will lead to the UK stumbling out of the club on bad terms, almost by accident.

After a week of posturing on both sides, Monday will see the second round of talks between Barnier and David Davis. Yet more than a year since Britain voted to leave, ministerial, official and Brussels sources told the Observer that the UK urgently needs to produce clarity on its demands in order to dodge a Brexit calamity.

Some of the most despairing insiders say that the government’s plan on some of the key issues is little further on than it was several months ago. While some blame a lack of Whitehall expertise or an overzealous Brussels making things difficult, most insiders point to British political paralysis.

Such is the frustration in some quarters in Brussels that during a recent trip there, one former Labour cabinet minister was stopped by a despairing commission official and asked: “What exactly is it that you want?”

Insiders point out a series of short, medium and long-term issues where the British position remains unclear. This week’s talks will concentrate on the short-term issues – demands by Brussels for “sufficient progress” to be made on the rights of EU citizens in the UK, the divorce bill and the Irish border. While pressing, sources say they are the most solvable.

It is understood that the Treasury is well placed in calculating what may have to be done to secure good terms. It will require, however, the government to be clearer that it agrees with the principle of coughing up the funds to cover its share of EU liabilities.

Boris Johnson’s call for Brussels to “go whistle” for the money caused despair, but on the eve of talks, the mood in Brussels is cautiously optimistic. Diplomats sense a change, after the UK quietly recognised financial obligations to the EU in a written statement to parliament on Thursday.

Having feared the talks were heading for a crash, the statement lifted the mood. “For the first time they have agreed we can talk about financial obligations,” said Elmar Brok, a veteran German centre-right MEP, who helps coordinate the European parliament’s response to Brexit. “Whether we agree on the result at the end of the day is a different story.”

Elmar Brok said progress has been made, although the outcome is uncertain. Photograph: Francois Lenoir/Reuters

In the medium term, the shape of the transitional deal which kicks in on the day after Brexit is still unresolved in the cabinet and causing serious concern on both sides. While some believed Davis was warming to the Treasury’s preferred option of a big deal lasting several years, including access to the single market, effective adherence of the customs union and a big fee, others believe he has cooled.

A post-election push by business to back such a deal succeeded in securing more meetings, but it is unclear to many of the executives involved whether they have been plugged into anything real. Even Sir Jeremy Heywood, the usually adroit cabinet secretary who survived the fall of the Cameron government, is said to have come with options for a future customs union deal that will not survive first contact with Barnier’s red pen. And while the Treasury is pushing plans for a soft transition out of the EU, they will come to nothing if the divorce bill issue does not progress.

Then there is the long-term Brexit deal, which is hampered by May’s continued stance that the European Court of Justice (ECJ) should play no role and the “hard Brexit” vision she has set out, including leaving the single market and customs union. While a deal on trade in goods looks straightforward to some officials, securing a deal covering financial services and finding a solution to extremely onerous non-tariff barriers remains completely unresolved.

Some ministers point out that the uncertainty is already causing business to plan for the event of a “no deal” scenario, fearing that a lack of flexibility will mean that such an outcome becomes a self-fulfilling prophecy. By the time any deal is reached, they say, the businesses most at risk will have already gone.

So what is causing the problems? There are two historic issues that officials point to as having dented the preparedness of the civil service. One is that it is paying the price for neglecting its trade and negotiating skills. “Civil servants rarely won praise from a minister for dealing well with Europe,” said one mandarin. The other is the failure to make any contingency plans for Brexit before the referendum, on the orders of the Cameron administration. On top of that, there is a natural capacity issue given the size of the task.

“Everything seems to be in silos,” said one minister. “Our industrial strategy seems to have gone a bit flat like a souffle. There should surely be industries that we decide are strategically important that are prioritised in any special deal we do, but I’m not hearing many voices on that. The risk is we’re defaulting into a pretty basic deal.”

Some ministers wonder whether there is still a reluctance among senior civil servants to speak hard truths to the key British players. Others say that May’s lack of clubbability and reluctance to engage with other European leaders has meant she has not been confronted with the reality of the EU’s concerns.

Yet most point to a lack of leadership – a weakened prime minister who, in the view of some inside Whitehall, is more reliant than ever on the right wing of her party for survival. Compromise has become a dangerous word. As far as some inside the machine can tell, she “remains on Plan A”, despite the hints of compromise from Davis and the chancellor, Philip Hammond.

A newly resurgent Europhile wing of the Conservative party is now trying to urge cabinet ministers to take a lead. Rather than wait for votes in parliament that show there is no majority for the Brexit model set out previously by the prime minister, pro-Remain cabinet ministers are being told to be brave and modify the offer to something the EU – and a majority in parliament – would accept.

“If the government engineers a situation in which its negotiating position is the same as what parliament is looking for, that isolates the headbangers and those who won’t compromise,” said one senior Tory MP. “I would lay it out to other members of the cabinet who take a different view, and say, ‘look, sooner or later we will lose a vote in the Commons. The election result means you can’t have the Brexit you wanted.’

“If we can’t do this in parliament, we will end up losing votes, confidence will be lost in the Conservative party, sooner or later something titanic will happen. We will have no credibility whatsoever and we will be thrown out in a 1997 situation.”

As one minister puts it, the government has two options. The first is to plough ahead with the current “hard Brexit” in order to show Tory Brexiters that it had been attempted, which could help keep the party together. The second is to begin making clearer concessions in crucial areas now – which are likely to unleash internal accusations of capitulation. “Either way it’s pretty horrid.” Even given all the challenges, the economic pain caused by talks collapsing means that most believe a deal will be done. There is some time before the crunch comes. The next crucial moment will be the Brussels summit in October, soon after the Conservative party conference. It is at that point the EU will judge whether initial talks over citizens, money and the Irish border have progressed sufficiently to start talking about the future EU-UK arrangements.

Optimists say there are small signs of progress. They point to signs from Downing Street and Davis that they are prepared to shift on allowing some kind of interim role for the ECJ, as well as a small acknowledgement that some payments could continue to the EU after exit. Others say it is simply too soon for the government to be making clear concessions, which could cause havoc in the Tory party.

In Brussels, relief over signs of compromise is tempered with uncertainty. EU insiders see a government that looks adrift and ill-prepared to sell the reality of Brexit to an electorate who were pitched a Global Britain, that could keep all the advantages of EU membership without having to play by the rules.

“The government is not in full control and this makes it difficult,” Brok told the Observer. “The most difficult thing is to reach a compromise. We are not sure whether this British government can sell any compromise at all … But that is the British problem, not our problem.”

Polish MEP Danuta Hübner thinks the British government’s “confusion” and “lack of unity” stems from the referendum campaign and the absence of any advance planning on Brexit.

“The British government still needs some time to establish a firm position on substance,” said the former Polish minister, who now chairs the European parliament’s constitutional affairs committee.

“The negotiations are an important learning process, not only for politicians, but also for citizens in the UK, because there wasn’t much explanatory discussion [and] there was a lot of confusion at that time [of the referendum campaign].”

EU insiders, however, now perceive more stability in the British government machinery. After early doubts about the role of Britain’s EU ambassador, Sir Tim Barrow, sources say they are reassured he is a player in the UK’s Brexit team.

Amid rising support for the EU and setbacks for Brussels-bashing populists at the ballot box, Europe is relishing the unity of purpose Brexit has created. However, diplomats look across the channel with curiosity and regret.

“It is not as if we go around high-fiving each other and saying ‘oh they are in such a mess’,” said one source. “Au contraire. There is no joy in this for us and it is not helpful either.”

THE KEY PLAYERS

