Under the Vienna convention the diplomatic bag is inviolable. That’s because states have long accepted that ambassadors need to be able to report home in frank terms without the prying eyes of the host nation reading what they write. The principle still applies, even though sealing wax and Queen’s messengers have been replaced by encrypted emails. But all the effort to keep diplomatic communications secure counts for nothing if they are leaked by the home side.

The scandal surrounding the reporting from British ambassador Kim Darroch in Washington is not that he was sending home his unvarnished analysis: that’s what good ambassadors have done for centuries. It’s that someone inside the British system deliberately amassed a stash of his assessments, then chose the moment of maximum impact to leak it. This was not a spontaneous decision to make public a single document: it required premeditation and therefore an agenda. Can it be a coincidence that the material appeared in the middle of the Conservative party’s leadership election?

Some have said Darroch should have been more circumspect. But that’s to undermine the whole purpose of having diplomats

Some have said Darroch should have been more circumspect about what he wrote. But that is to undermine the whole purpose of having diplomats overseas. Their job is to give their best judgment and advice. It is up to ministers whether to take it or not. The quality of foreign policy decisions would certainly suffer if ambassadors censored their reporting for fear of it being leaked.

The British diplomatic service has long been regarded as one of the best in the world. That’s partly because it employs excellent staff, trains them to be shrewd analysts and encourages them to give their views without fear or favour. It’s also because Whitehall has had a culture of sharing information efficiently among those who need to have it. When I was serving abroad, I was often better informed about meetings between heads of government, for example, than the officials in the local foreign ministry, where information was regarded as power and tightly held.

Given the range and complexity of risks facing modern governments, the capacity to integrate information from a wide range of sources is more crucial than ever to good policymaking. If or when Britain does finally leave the European Union, negotiating a new basis for relations with our former partners and rethinking national strategy will put unprecedented demands on Whitehall. It will be essential that ministers get the best advice from their civil servants, and that decisions are based on pooling all the key information.

Play Video 1:34 Jeremy Hunt says Trump administration is highly effective – video

That can only happen in an environment where everyone has trust that confidential information will remain just that. It’s embarrassing that we all know Britain’s ambassador believes the US administration is “dysfunctional”, “unpredictable”, “faction-riven” and “inept”. But the most toxic aspect of the latest leak is that it undermines trust in that confidentiality. Following the briefing to the media of discussions in the National Security Council about the role of Huawei in the UK’s 5G network, it suggests a deeply worrying pattern is emerging. Some people in the system are abusing their access to national security information to pursue political goals without any thought for the damage to the county’s interests or reputation.

I do not believe the content of Darroch’s messages will do any lasting damage to the British-American relationship. Of course it’s not a great day for ambassadors to wake up and find their reports on the front page of a newspaper. It never happened to me, but there were plenty of occasions when I pressed the send button and then thought: “I really hope that doesn’t leak.” The relationship with Washington is based on strong and deep shared interests. Those are unchanged by the leaks. After all, the Americans had 250,000 state department cables from their embassies all around the world revealed by WikiLeaks in 2010 without lasting damage to their foreign relations.

No, the damage is elsewhere – in three areas. First, to Britain’s reputation as a country that knows how to keep its secrets, with the suspicion that the divided and antagonistic state of our politics is spilling over into the handling of sensitive information.

How can Kim Darroch represent Britain in Trump’s Washington now? | Simon Jenkins Read more

Second, to the willingness to share information efficiently around the system so that good decisions are taken on the basis of up-to-date knowledge of the true state of things. A natural reaction to what has happened might be to prune distribution lists, and to apply the “need to know” principle even more tightly. If taken too far, that could weaken one of the strengths of the British administrative system just when we will need it to be at full effectiveness. Third, the damage is to the confidence of civil servants to put their frank thoughts to ministers. They will have seen Ivan Rogers hounded out of his post as permanent representative to the EU after some of his reporting was deliberately leaked to undermine him. Now this.

This pattern needs to be broken. I hope Darroch will have the staunchest backing from the future as well as the current government. I hope he will be replaced with another professional when his tour ends. And I hope that the leak inquiry will be pursued with real determination, and will involve the police if there is reason to believe the Official Secrets Act has been violated. The person who carried out this premeditated effort to undermine our ambassador, and with it our national interest, needs to be very worried today. When they are caught, it should be a career-ending event. That’s the only way we can stop the rot.

• Lord Ricketts is a former permanent under-secretary at the Foreign Office and ambassador to France