Britain has not made the running on any foreign policy issue since at least 2013 and suffers from a fundamental absence of strategic thinking about its role in the world, the most senior foreign policy adviser to David Cameron has said.

John Casson – who worked in No 10 from 2010 to 2014 before becoming UK ambassador to Cairo until last year – warned of a lack of confidence across Whitehall about Britain’s role in the world. “We see lots of actors smaller than us by most metrics making more of a difference because they have a clearer-eyed sense of their strategic intent and the power they hold,” he said.

Speaking at a debate on future British foreign policy at Chatham House, he also lamented that no British prime minister since either Tony Blair or Gordon Brown had thought it worthwhile to invest in personal relations with foreign leaders, adding this judgment had cost Britain dear.

Casson criticised Boris Johnson, describing the prime minister’s strategy as haphazard. He said: “Boris Johnson makes speeches about the cult of the strongman and wanting to stand up to human rights abusers in certain situations, but then in other situations pops up making contradictory signals to that.”

Without naming him, he also delivered a scathing assessment of a recent foreign secretary’s ability to think strategically.

Casson recounted: “I once got stuck in traffic in the back of a car with one of the most recent foreign secretaries and I said: ‘Given all of us in the Foreign Office spend our time trying to make sure you don’t do anything different from what we normally do, what is it the thing that you think you are trying to get us to do to leave behind as your legacy?’

“This person thought about it for a bit and said: ‘Well, I really think we should be trying to save the African elephant and leading on the international wildlife trade, and it is really important that we bring girls into school worldwide and I suppose one of these conflicts in the world, we should be taking a lead on solving.” We then had a discussion about whether Palestine, Libya, Syria or Yemen was particularly ripe for British intervention.”

Johnson repeatedly as foreign secretary highlighted the future of elephants, using one speech at Chatham House to admit he was “obsessed with the tragic fate of the African elephant”. He also said more girls in education was the key to unlocking some of the world’s greatest problems.

Casson added: “I say this by way of saying there has been an absence of any depth of strategic thinking not just at political level but also confidence amongst officials not just in the Foreign Office but across Whitehall to think clearly about what is the nature of the power that we hold.

“The bald fact for me is British foreign policy – as I was brought up to recognise it – came to an end in September 2013, when David Cameron made a conscious decision to stop being an activist prime minister on foreign policy after he lost the Syrian vote in the Commons. Since then it is very difficult to name an issue on which Britain has made the running in the international space.”

He insisted this was not a lament for liberal interventionism, but instead a breakdown in a pro-active optimistic foreign policy drawing on Britain’s strengths.

As a former foreign affairs private secretary to Cameron, he added a prime minister “needs to be on the telephone a lot and travel a lot and see a big part of their job as having relationships with foreign leaders for the sake of relationships with foreign leaders. Since Tony Blair, and perhaps Gordon Brown, we have not really had a prime minister that has had the capacity and the appetite to see as a fundamental slice of their job investing in those political relationships. It has cost us in terms of foreign policy outcomes.”

He said from his own experience at No 10 that it is “a bloody battle” to get the prime minister to focus on foreign leaders since “the building is full of people who think it is a waste of time because it does not win you elections”.

He attributed the decline in UK foreign policy influence to a range of factors including the absence of the US holding the ring for its allies, the distraction of Brexit, the shift to a multi-polar world, but also the absence of any confidence in the British political system about what the UK can offer to help the shape the world.

The critical factor in shaping British future power would be Brexit, he said, saying the world would be watching and smelling for the signs of economic success or failure. “If the rest of the world think we are in a mess, the rest of our foreign policy will be very difficult to conduct.”

He added that Britain had confused its international brand. “Are we an open country that aspires to set an example and stand up for certain type of norms and influence?” He said this showed in the country’s policies on visas, foreign investment and universities. “It is not just about foreign policy statements. We need to find our way back to a clearer narrative of what is our point of attraction.”

He also urged Britain to be ruthless about aligning its future leverage after Brexit, saying it needed to think whether it wished to remain close to Europe, become the best friend of the newly emerging nations or instead maintain as long as possible the transatlantic umbrella. He said the few bureaucratic changes under way in Whitehall to prepare for Brexit did not match the scale of the challenge facing the country.