The three cabinet ministers in charge of Brexit – Boris Johnson, Liam Fox and David Davis – resemble “three blind mice” stumbling around the world with inconsistent messages on how to leave the EU, a Conservative former minister has said.

Nick Herbert, who led the Conservative remain campaign during the referendum, joined a number of senior Tories who are warning that there are still severe risks to the economy as government ministers try to work out a Brexit plan. He said it was essential to accept the result but warned against a “naive ideal of a new Britannia” making ministers overconfident that they will secure a good result for the UK.

“Conservatives must beware Brexit fundamentalism, or giving themselves up to a romanticised 1950s vision of Britain, a country of imperialist chauvinism,” he wrote in an article for the Guardian.

The first day of Conservative conference was dominated by Theresa May’s pledge to start the process by March next year, delighting Tory delegates at the International Convention Centre in Birmingham. She faced criticism about the competence of her three senior cabinet ministers in charge of Brexit and warnings against pursuing a “hard Brexit” that sacrificed access to the single market in order to achieve controls on immigration.

Herbert, the former minister for policing and criminal justice, said the government needed to be careful not to be swayed by the most hardline Brexiters and warned that the jury is still out on the performance of Johnson, the foreign secretary, Fox, the trade secretary, and Davis, the Brexit secretary.

“We should be talking about financial passporting and the need to prevent a haemorrhage of banking jobs from the City, not fixating on the colour of our passports. We should be discussing how to strike the best deal with our biggest trading partner, not how to relaunch a royal yacht,” he said.

Theresa May faced criticism about the competence of her three senior cabinet ministers in charge of Brexit. Photograph: Matt Cardy/Getty Images

“The so-called ‘three Brexiteers’ have so far rather more resembled three blind mice, stumbling around the world’s capitals with inconsistent messages, united only in their assurance that it will be alright on the night,” he added.

“Now the politicians who were already inclined to be deaf to business concerns have been emboldened by the apparent resilience of Britain’s economy in the face of a predicted short-term shock. The risk is that important concerns about the long-term impacts of a hard Brexit will be dismissed. We may all be Brexiteers now, but leaving without being able to reassure inward investors or the financial services industry that they will be able do business as usual would be a catastrophe.”

Herbert, the MP for Arundel and former Home Office minister under Theresa May, is now involved in the Open Britain campaign group holding the government to account on leaving the EU. He said there was a “staggering” collective naivety about the ease of doing global trade deals, when these typically took years to negotiate and were “not exactly flavour of the month” with the public when it came to agreements like TTIP. “One senior minister said there would be a trade deal with New Zealand before Christmas,” he said.

Another leading Brexit campaigner predicted a deal with India “faster than you can say masala bond”.

“Those who talk so excitedly do not stop to consider the inevitable first demand of major agricultural exporters, which is that we drop our farm subsidies. And they clearly haven’t understood that even poor deals take years to complete.”

However, such warnings, and others about Brexit from former cabinet ministers Nicky Morgan, Ken Clarke, Anna Soubry and Dominic Grieve, did not dampen the mood of the first day of May’s first conference as prime minister, which appeared optimistic about the prospects for Britain after leaving the EU.

At a fringe event, Fox, who returned to the cabinet as trade secretary under May, was applauded as he spoke with tears in his eyes about his pride in the UK for voting to leave the EU.

Liam Fox, the trade secretary, on Thursday. Photograph: Anthony Devlin/AFP/Getty Images

“I have never felt so proud of my fellow countrymen and women as I did at that moment. I thought: ‘You have taken on all the doomsayers, you have really had faith in what Britain can do, you really believe in this country and yourselves.’

“No matter what happens to me in politics, that will always be the single moment I remember the most,” he said. “That was history being made and to just be a little part of that, I can’t even begin to describe how that felt.”

Fox also denied there were any tensions with either Davis or Johnson as has been reported, saying he was an admirer of both and worked “very, very closely” with them. He said Davis had “one of the best strategic minds in politics” and Johnson was a “very clever individual who has thought a great deal” with a huge understanding of Europe.

The battle between those who want to minimise the separation with Europe and others who want a clean break was mostly played out on the airwaves rather than the conference floor.

Iain Duncan Smith, the arch-Eurosceptic former cabinet minister, dismissed all fears about the economic consequences of Brexit, saying car manufacturers in the UK would not be “affected adversely” in any shape or form by the decision to leave.

This was challenged by Soubry, who was involved with founding Open Britain and backed the remain campaign, who told ITV’s Peston on Sunday that it was “rubbish” to suggest the future trading relationship with the EU would be as favourable as now.

“We’re going to get something worse, obviously we are, and we don’t hold the cards, the EU does,” she said.

Tory modernisers delivering such warnings were vocal but in short supply, with May enjoying the backing of most backbenchers as well as remain supporters who have joined her frontbench.

David Cameron and George Osborne were notably absent, although Lord Feldman, the former party chairman, was one of the few Cameroons spotted in the crowd.

Conservative delegates in the main hall gave May, Davis and Johnson a rousing reception, while applause for the legacy of David Cameron was more muted.

Dozens of fringes each day at the four-day conference are dedicated to the details of leaving the EU, from the impact on relations with Turkey after warnings from Brexiters against it joining the EU to the future for forestry post-Brexit.

One cabinet minister told the Guardian he was not bothering to make any big policy announcements in his speech because the whole conference was so dominated by the subject of leaving the EU.

Some Conservative remainers were, however, still pining for pre-Brexit times. Flick Drummond, a Tory MP who backed the remain campaign, said she was “still going through the mourning stage” of the Brexit vote.

“It’s more about the influence we’ve lost within Europe rather than anything else. I now feel we’re really on the outside of EU governance,” she said. “But we’ve got to make the most of it. We have to get out there very quickly now.”