Amid the prosaic setting of British politics, with its fusty civil servants and turgid party meetings, the secretary of state for foreign and commonwealth affairs stands out as the most glamorous of cabinet appointments. There’s the exotic travel, the hobnobbing with dignitaries, the adrenalin rush of serious global events – who cares about the campaign to stop the local post office closing down when you’ve got a vote at the UN security council to deal with. And there’s the sense, bolstered by your Foreign Office staff and the never less than precarious condition of international relations, that what you say and do actually matters.

Still, behind all the photocalls, the joint accords and bland diplomatic statements, what does a foreign secretary really do? After all, when it comes to the most critical foreign policy – military intervention – it’s the prime minister who’s in the hot seat. Although foreign secretary is one of the three great offices of state beneath that of the PM, it doesn’t enjoy the autonomy of chancellor or the isolation of home secretary – a ministry from which prime ministers are often all too keen to distance themselves. At the key moments, foreign policy is the preserve of No 10.

Furthermore, at this present juncture, the job description of foreign secretary seems more amorphous than ever. Britain stands at an uncertain crossroads. The world order is shifting, and our traditional ally, the United States, is run by a maverick reality TV star who wants to pull back from international commitments while conducting global policy via his Twitter account.

Then, of course, there is Brexit. The nation seems stuck at the door, neither ready to face the harsh world outside nor prepared to return to the embittered marital home. For all the talk of getting our country back, there is a growing fear that we will return to being what John Updike once called “a soggy little island huffing and puffing to keep up with western Europe”.

And where in all this trepidation is the current foreign secretary, Boris Johnson? He appears to have been more vocal about the plight of the National Health Service – not traditionally seen as part of the Foreign Office’s remit – than he has been involved in the most momentous change in foreign policy since the second world war. And when he does intervene, as shown by his recent comparison of the Irish border to the line between the London boroughs of Camden and Islington, he tends to make more headlines than headway.

The real lessons of our failures in foreign policy over the last 20 years or so have still not been learned David Owen

In an age of American unpredictability, Russian subversion, Chinese expansionism and European divorce, what should British foreign policy be, and what does or should a British foreign secretary do?

To get an answer to these questions, I set out to speak to as many former foreign secretaries as possible. There are 10 of them still living. Philip Hammond, who was in the role from 2014-2016, is still in the cabinet, and therefore not forthcoming enough for our purposes. John Major only did the job for three months, barely enough time to unpack the family photos. Douglas Hurd, who retired from politics two years ago, is 87 and has recently been unwell, and Peter Carrington, the last hereditary peer to hold the post, is 98, and a little out of circulation.

That left six who were keen to talk, some of them very keen, so much so that it required all my powers of diplomacy to extract myself from their loquacious company.

There aren’t many places to go when you’ve been foreign secretary. Either you become prime minister or it’s down or out. In theory, you could get shifted sideways to chancellor, as happened with Hammond, but in the case of the six I spoke to, foreign secretary was the highest political office they were to hold.

So I couldn’t help but detect a melancholy hankering for grander times enshrouding some of the reminiscences. Even David Owen – who at 79 still displays the intellectual confidence that intimidated many political opponents, not to mention allies – betrayed a certain nostalgia for his globetrotting days in office. He speaks in forensic detail of meetings and conversations that took place over 40 years ago, suggesting they still occupied a treasured place in his thoughts.

I spoke to Owen at his home, the location in 1981 for the “Limehouse Declaration” that launched the Social Democratic party, with its commanding view of the Thames.

Owen became foreign secretary at the age of 38, one of the youngest in history. He served James Callaghan’s government between 1977 and 1979. Was he daunted by the size of the job at such a young age?

“I was always accused of either being abrasive or arrogant. Which in part was true,” he says in a tone that does not convey regret. “But, you know, you have the confidence of youth. One of the things about being appointed young is that you can absorb a huge amount of information quickly. And it isn’t long before you have to keep up to date with the constant flow of telegrams.”

Owen believes that he was the last but one of what he calls “the historic foreign secretaries”, those who ran foreign policy themselves rather than meekly deferring to the prime minister of the day. “I think the job changed hugely after [his successor] Carrington’s resignation [at the start of the Falklands war],” he says.

Thereafter, he believes an emboldened Margaret Thatcher took ever more control of foreign policy. This trend, according to Owen, slackened a little under John Major but was redoubled by the time of Tony Blair. “I mean, [Blair] didn’t really like Robin Cook. He didn’t like any of his foreign secretaries.” He says Labour’s most successful prime minister subsumed much of the foreign secretary’s remit “into the No 10 machinery” in what he describes as “an unbelievable constitutional power grab”.

Quick Guide Boris Johnson's errors of judgment Show Nazanin Zaghari-Ratcliffe Boris Johnson said that the British-Iranian citizen Nazanin Zaghari-Ratcliffe, convicted of spying in Iran, was “simply teaching people journalism” – a statement her family and her employer both said was untrue. His comments were subsequently cited as proof that she was engaged in “propaganda against the regime”. 'Dead bodies' After Johnson suggested that the Libyan city of Sirte might become a new Dubai once “the dead bodies” were removed, Downing Street said it was not “an appropriate choice of words”. Myanmar The foreign secretary was accused of “incredible insensitivity” after it emerged he recited part of a colonial-era Rudyard Kipling poem in front of local dignitaries while on an official visit to Myanmar. Whisky sour Johnson apologised after causing a “livid” reaction in a worshipper in a Sikh temple in Bristol by discussing his enthusiasm for ending tariffs on whisky traded between the UK and India. Alcohol is forbidden under some Sikh teachings. Continental drift Boris Johnson referred to Africa as “that country” in his Conservative party conference speech.

Tweet like Trump The foreign secretary suggested he wished he could tweet like Donald Trump, despite intense criticism of the US president’s use of Twitter, on which he has launched personal attacks against his foes.

Unsurprisingly, the other five foreign secretaries I spoke to didn’t agree that they were their PMs’ ventriloquist dummies. All acknowledged Owen’s general point but in one way or another saw themselves as exceptions to that rule. Perhaps the most confusing case is the current foreign secretary. He appears to have more political liberty than any cabinet member in a generation… except, it seems, in matters of foreign policy.

What is Owen’s opinion of the man? “I’m not going to answer this question,” he replies, uncharacteristically coy. “I joke and say that it’s a trade union of foreign secretaries. It’s a hard enough bloody task without being sniped at by your fellow foreign secretaries.”

Later on he says he thinks Johnson is a “big figure” and “perfectly competent”. So I ask about the gaffe involving Nazanin Zaghari-Ratcliffe, the Anglo-Iranian woman imprisoned in Tehran on charges of “plotting to topple the Iranian regime”. In evidence to the foreign affairs committee, Johnson controversially described Zaghari-Ratcliffe as someone who was “teaching people journalism”, a phrase that was jumped on by the Iranians as confirmation of her supposed treason.

“I don’t believe he made a mistake on this because he’d forgotten,” says Owen. “Now, either he is not devoting enough time to reading and keeping up to date in this huge field, or the selection of telegrams for him is not good enough from his own private office.”

I ask a few more Johnson-related questions, and the fiercely pro-Brexit Owen concludes with a firm warning: “All I’m telling you is that there are ominous signs that the real lessons of the failures of British foreign policy over the last 20 years or so have not been learnt: the prime minister has got to appoint somebody as foreign secretary they trust. The foreign secretary has to have a dominant relationship over the secretary of state for defence, but a close one. But ultimately those two need to work together, and there must be a readiness of the prime minister to support them in battles with the chancellor wherever humanly possible.”

Malcolm Rifkind, foreign secretary 1995-1997, in his Westminster flat. Photograph: Suki Dhanda/The Observer

Malcolm Rifkind, who served under John Major’s premiership, largely endorses Owen’s stark appraisal. I meet him in his London flat a few hundred yards from Westminster. The place is thick with photos of Rifkind with major figures like Margaret Thatcher and Mikhail Gorbachev, and the wistful air of bygone glories. Rifkind’s career came to an ignominious end after a “cash for access” sting operation by the Daily Telegraph and Channel 4 led him to step down as chairman of the intelligence and security committee and give up his safe Kensington seat. A parliamentary investigation later exonerated him.

He became foreign secretary in July 1995, almost exactly in the middle of that misleading lull between the fall of the Berlin Wall and the collapse of the Twin Towers, when history was alleged to have ended. But history was stubbornly continuing in the Balkans. A week after Rifkind got the job, the massacre of more than 8,000 Muslim Bosnians took place at Srebrenica.

Rifkind’s predecessor, Hurd, pursued a policy of neutrality, supported by Rifkind, that was criticised for leaving the Bosnian Muslims at the mercy of well-armed Serb nationalism. Srebrenica stung Nato into action, and the air strikes that followed brought the Bosnian war to a swift end. After the failure of the non-interventionist line to impede the war, the belated bombing seemed to herald the new paradigm of progressive military engagement that Tony Blair would later adopt.

Rifkind demurs. “I think that the road to Baghdad began in Kosovo,” he says sternly.

Back in the 90s, such was the sense of optimism that even the Israel/Palestine conflict seemed soluble. Rifkind believed that peace was within reach until Yitzhak Rabin was assassinated by an Israeli rightwing extremist in November 1995. The day after Rabin’s funeral, Rifkind visited the Palestinian leader, Yasser Arafat, in Gaza. “He said, ‘I’m sorry, this is going to be a shorter meeting than I would’ve liked because I’m being driven up to see Leah Rabin (Rabin’s widow), to pay my condolences privately.’ The chemistry was of a totally different order to what we have now, and Mahmoud Abbas [the current Palestinian leader] is a much more moderate guy than Arafat ever was.”

Rifkind tells the story to demonstrate how critical personal relations are to international relations. “Where people are looking for solutions, they find solutions. When they’re looking for problems, they find problems. You can make a difference by your own personal involvement, the relationship you have with the foreign ministers of other countries, even if they don’t always agree with you.”

In Boris Johnson’s case this task has been made more challenging by his track record of insulting foreign leaders. Just two months before he became foreign secretary, he wrote a poem about President Erdoğan of Turkey having sex with a goat. Four months later he was in Ankara meeting Turkey’s EU minister, Ömer Çelik, and promising, now that Britain was leaving, to support Turkey’s bid to join the EU.

Notwithstanding these lyrical contributions, Rifkind believes Johnson is able to build good personal relationships but suggests he’s undermined because he lacks the backing of Theresa May. “I’d have made David Davis [secretary of state for exiting the European Union] part of the foreign office structure – but then I would have been more confident in the choice of foreign secretary that I had.”

Before I leave him, I ask Rifkind what he thinks of his successors. He runs through them in chronological order. Robin Cook, he says, was intensely disliked within the foreign office because of his chaotic style of working, but was held in high regard by foreign ministers.

Cook notably declared his aim to have an “ethical dimension” to policy, a statement on which Rifkind took issue. “Not because there shouldn’t be an ethical dimension, but to imply somehow you can only have as your diplomatic allies people who share your political values just doesn’t make sense in this sad, imperfect world, and ultimately he agreed with that.”

Jack Straw, who followed Cook, he describes as a “highly able guy” who “was perhaps unfortunate having Blair as his boss”. Straw lives in a quiet square a few minutes walk from the Oval cricket ground in Kennington. He fell foul of the same sting operation as Rifkind and stood down from parliament before the last election. He too was exonerated by parliament’s commissioner for standards. He’s a genial character who, despite having been home secretary and foreign secretary for a combined total of nine years, displays none of the sense of entitlement that politicians of a similar standing tend to exude.

Straw believes foreign secretary is a job apart because, unusually in British politics, foreign policy is “broadly bipartisan”. He was by his own account quite surprised to be given the job by Tony Blair, with whom, he says, contrary to Owen’s claim, he had a good relationship. “The first thing you’ve got to accept if you become foreign secretary is that the lead on foreign policy is going to be taken by the head of government. It was ever thus.”

Jack Straw, foreign secretary 2001-2006, in his home office, London. Photograph: Suki Dhanda/The Observer

I mention that Owen said it was different in his day. “Oh, I enjoyed significant autonomy as well on some stuff, most notably about Iran. I’m not trying to diminish David’s role, but I think it’s more true with Peter Carrington over Zimbabwe, where he just ploughed on when Margaret Thatcher was in a relatively weak position in the early stages of her premiership.”

He says the point is that while diplomacy is about resolving international disputes without resorting to war, the “moment that the issue of military action appears on the horizon, the centre of gravity is bound to move to No 10. That’s been true at all times.”

He started at the Foreign Office in June 2001. The Balkans had settled down, and the first three months were almost becalmed. His priority was to curb the excesses taking place in Zimbabwe by helping financially with the land transfers from white farmers to black “war veterans”. Straw says he barely consulted Blair over this.

“And then,” he says drily, “9/11 happened and everything changed.”

During the course of the following 18 months, the UK, in support of the US, would enter two far-reaching wars, first in Afghanistan and next in Iraq. There would be mass anti-war protests, disputes about legality, allegations of British involvement in torture and renditions, British citizens held in Guantanamo, the spread of radicalisation across the Muslim world, the war on terror and instability throughout the Middle East. Short of a world war, no British foreign secretary is likely to experience that kind of succession of events for the foreseeable future. The policy of intervention that had secured peace and stability in Kosovo and Sierra Leone had, by the end of Straw’s stint in office, begun to look deeply suspect.

It was Blair who made the decisions to go to war, but Straw was in agreement. “In practice, we had no option with Afghanistan, unless you wanted to leave it to the venal forces of the Taliban and their readiness to allow al-Qaida to carry on with its training camps and to commit atrocities around the world.”

About Iraq he was much more sceptical. His wife and children were against the war and would have joined the protest march had he not been foreign secretary. He knew that soldiers and civilians would be killed in significant numbers, although he never expected the scale of bloodshed that was to follow.

“That’s a responsibility you’ve got to bear,” he says, “and if you don’t, you shouldn’t take on these jobs.”

Right up until the last moment, he was considering telling Blair that they should offer “moral support to the Americans, help with reconstruction”, but not put troops on the ground. In the end, he was persuaded by Saddam’s intransigence with weapons inspectors. “I thought they must have [WMD], otherwise why are they dicking about refusing to let the inspectors in.” The Iraq war remains his biggest regret in office. With hindsight, he says, “it would have been better that we’d not started”.

In reality, Straw’s influence was limited. He had good relations with his American counterpart, secretary of state Colin Powell, but Powell was himself sidelined by others in the Bush administration. The only effective power Straw had was to resign, and by the time he realised that a civil war was brewing, his resignation would have had no impact on policy.

Which brings us to the much vaunted, “special relationship” with America. Owen had told me that in his opinion it operated only in military and intelligence terms. But if the British military was subordinate to woeful American planning – the disbanding of the Iraqi army and dismissal of all Ba’ath party members was, says Straw, a disaster – then how special was that arrangement? “The special relationship was a phrase I sought to avoid,” he says, “because I thought it was patronising of the Americans and a piece of conceit by us.”

Even after two wars and an unrelenting workload, Straw was disappointed to be removed from the job by Blair. “It was very odd the day after,” he says. “Enoch Powell famously said that all political lives end in failure. He was wrong. The thing is, all political lives end – and they end rather abruptly.”

But what of that political life that, like a cat, appears to have multiple extensions? What does he think of Johnson?

“He’s got a good grasp of history,” he says evenly. “He understands about political diplomatic forces, which is one of the most important qualifications for being foreign secretary. But sometimes he lets his persona as a showman get the better of him. I personally don’t think that works very well.”

Margaret Beckett, foreign secretary 2006-2007, in her office at parliament, where she has been Derby South MP for 35 years. Photograph: Suki Dhanda/The Observer

Straw was replaced by Margaret Beckett, the first ever female foreign secretary. I visit her at her capacious office in the bowels of parliament, where she still sits as MP for Derby South, 35 years after first being elected. Rifkind described Beckett as “a bright, intelligent lady” who didn’t have “much impact during the short period that she was there”. “It wasn’t,” he felt, “her scene.”

It seems that Beckett wasn’t welcomed by some in the rather clubby atmosphere of the Foreign Office. She was told by a female Foreign Office veteran soon after she arrived: “I hope you realise there are people here who don’t think a woman should be foreign secretary.” She was taken aback by this news, coming as it did six years into the 21st century. Her approach to the job was to establish what the priorities were for foreign policy, but in this task she found the Foreign Office reluctant to help. “In the end,” she says, “basically I and my special adviser did it together.”

She believes her task wasn’t helped by her refusal to cultivate the media, and therefore, she believes, she was saddled with a difficult image. “Going around the gossip circuit, telling spiteful stories against my colleagues has never been something I thought was part of the job,” she says with a knowing smile. Instead she focused on developing good working relationships with her foreign counterparts. And in this, she did believe in the “special relationship”. “It was real, it was important, and it wasn’t particularly a problem for me. I mean, I had a good relationship with Condi [secretary of state Condoleezza Rice), and it wasn’t one where I just did what she thought I ought to do. We had a genuine dialogue.”

She never had to contend with the protests her predecessor witnessed but did receive flak for not calling for an immediate ceasefire when Israel went into Lebanon. She was even collared in a supermarket over Israel’s response to Hamas rocket attacks. “I remember someone shouted out, ‘Shopping while Gaza burns!’. And I thought, well, you know, we do have to eat.”

A cool head and resilience, she says, are the main qualities required for the job, and the “recognition that you will be blamed”. How would she have dealt with building a relationship with Rex Tillerson, Donald Trump’s secretary of state? “It must be very difficult because you sometimes get the impression the president doesn’t know where he stands from one day to the next.”

Like most of the former foreign secretaries I speak to, she believes that the current one should be playing a “much more major role” in the Brexit negotiations, and would be if it was anyone else but Johnson. It sounds, I say, like she’s not a fan. “When I was told that some of them in the Foreign Office didn’t think there ought to be a woman in the job, I thought to myself, I know exactly what sort of person they think should be foreign secretary. It should be somebody who went to the right school, who went to the right university, who is regarded as an intellectual and has wide knowledge of the world. And that’s what they’ve got.”

She deliberates before putting these thoughts on record because she thinks it’s a bit “bitchy”, but then goes on to call Johnson a “walking embarrassment” for his pronouncements on Brexit.

David Miliband, foreign secretary 2007-2010, at the International Rescue Committee HQ in New York. Photograph: Mike McGregor/The Observer

Beckett was in the job for only a year before she was replaced by the man everyone was expecting to take over, the man Rifkind described to me as “very, very good” with a “natural empathy” for the job: David Miliband. Now based in New York, where he runs the International Rescue Committee, Miliband spoke to me by phone. Foreign secretary in Gordon Brown’s government for three years from 2007, he says he had a clear vision of Britain’s role in the world when he got the job. “I talked about Britain as a global hub, connected to all the alliances that matter,” he says, and – once again contrary to Owen’s assertion – he insists he was his own man.

“Gordon Brown was avowedly a prime minister who wanted to spend less time on foreign affairs than his predecessor. And that was symbolised in the toe-curling day when he didn’t come to sign the Lisbon treaty – I beg your pardon, he arrived much later in the day than everyone else. When all the prime ministers and presidents were called upon to sign the treaty in a very large cathedral in Lisbon, I trotted up there on my own.”

He retains an engaged interest in global diplomacy, and it’s his assessment that there is now profound uncertainty afoot about Britain’s international role. “The abdication of American leadership on one hand, Brexit on the other, and all the causes and consequences of both those phenomena mean that it is legitimate to say there is an unprecedented questioning of Britain’s foreign policy.”

People look to Britain to be “pragmatic, sensible, stable”, he says, but that’s not the impression the world currently has. “It’s hard to know what British foreign policy doctrine is at the moment. The government is consumed by dealing with Brexit, even if the Foreign Office has been banned from negotiating it – and that’s a remarkable situation. There’s also the reality that Boris Johnson has done a better job of undermining himself than any attack from me or anyone else could do.”

But isn’t the post-Brexit vision supposed to be all about forming relationships with countries beyond Europe, reaching out to parts of the world to whom we’d previously undersold ourselves? “Obviously you can say there’s a global Britain under Brexit, which is the government’s argument, but there’s no question among the people I talk to in foreign policy circles and more widely that Britain has taken a turn towards isolation.”

So what should a foreign secretary be seeking to achieve in a post-Brexit world? “The job of the foreign secretary will be to try to recreate some of the structures and relationships that existed within the EU,” he gloomily concludes, “except with the very difficult fact that he or she is excluded from all the key meetings.”

William Hague, foreign secretary 2010-2014, at the Royal United Services Institute (RUSI), Whitehall, London. Photograph: Suki Dhanda/The Observer

Now in the Lords, William Hague, foreign secretary between David Miliband and Philip Hammond, paints a different picture of Johnson and Brexit. His pristine office is in Millbank, just along from parliament. He wants the UK to retain close links with the EU but argues that Johnson’s main job lies elsewhere. “I think he’s a brilliant guy,” he says, measuring his words. “He sees his job is to make sure Britain is engaged with the rest of the world while Brexit is going on. That’s critically important, not only in being innovative in finding new ways to cooperate with the EU when we’ve left, but also to show we’re not becoming less active in the world.”

By Rifkind’s estimation, Hague will not be remembered “in terms of policy”, but rather as someone who attempted to revive the Foreign Office, reverse the closure of embassies and restore foreign language training.

But Hague will also be remembered for the Libyan intervention – David Cameron’s call – and the failed House of Commons vote for intervention in Syria, following its use of chemical weapons. He describes this defeat as his “worst point” as foreign secretary, an “abrogation of this country’s responsibilities that made things in Syria worse and emboldened Russia to inflict more horrors on Syria than would otherwise have been the case”.

He doesn’t regret the removal of Colonel Gaddafi in Libya, even though that country is now divided among warlords. But he says, a little ruefully: “One of the things we’ve learned in the last 10 years is how difficult it is to create democracy in other cultures, and that doing that quickly can actually set things back.”

To return to Cook’s “ethical dimension”, I ask about the responsibilities of dealing with countries with bad human rights records. Do we not overlook such problems with, for example, Saudi Arabia, simply because we have lucrative trade deals?

“We do have commercial and myriad of other links with countries whose system of government we don’t agree with, but that does not mean we cut them all off, and, indeed, we would be in a terrible situation if we did.”

His policy, he says, was always to raise human rights at senior level meetings. “I think part of the necessary skills of foreign secretaries is to be frank and civil at the same time.”

Yes, but should we be arming Saudi Arabia? He gives the kind of answer that can only be described as diplomatically evasive. “It’s clearly in the interest of our national security for Saudi Arabia to be a strong and successful nation and for its leadership to be successful, and it’s definitely in the interest of British industry for it to be able to buy from British industry. But it is also engaging in conflicts that raise questions about the use of what we sell to them.”

It was John Kenneth Galbraith who said that the one reliable rule of diplomacy is that when an official says that talks were useful, you can safely conclude that nothing was accomplished. And in the end, the job of foreign secretary, for all its superficial glamour, is something of a thankless task.

You spend most of your time and effort working behind the scenes for trade, peace, better relations, ideally making the right noises on human rights, and no one takes much notice until you make a mistake or your prime minister decides to send the air force in.

And yet, although Boris Johnson once called being mayor of London “the best job in British politics”, most of his predecessors as foreign secretary would say that title belongs to the one he has now.