There's more to the WikiLeaks dispatches than leaks. Look behind them, at the writers, and you see the loyal rearguard of America: an imperial power in retreat.

There was a tradition in our Foreign Office that a retiring ambassador could blow off steam. In a final, exuberant telegram to Whitehall, he could say exactly what he thought of the country he was leaving, and of the folly of the Foreign Office in ignoring his advice.The best telegrams were treasured by young diplomats. But they began to leak into the press. And a few years ago this privilege was suppressed.

Now the WikiLeaks eruption has smothered the world with the secret thoughts of the state department's ambassadors. Tomorrow's Observer, focusing on China, reveals fascinating data about Chinese "muscle-flexing, triumphalism and assertiveness" (as the US ambassador put it). But with the cables comes a snapshot of the state department itself. It's a unique window on America's search – with diminishing confidence – for a coherent, inspiring account of what the US is trying to achieve in the world.

These diplomats who didn't want us to know their thoughts are not mere cogs in an imperial machine. Many emerge as wise, courageous, patient, likeable men and women– especially the women, who lead so many US embassies. Their view of their host countries is not rosy. You begin to absorb their vision, in which America is the only adult in a world of grasping, corrupt, unreliable teenagers who cannot be abandoned to their own weakness.

The test of an ambassador is telling truth to those who wield the power – having the guts to tell the department that its plan is a delusion. Here is Anne Patterson in Islamabad, discussing Pakistan's support for "terrorist and extremist groups" and telling Washington "there is no chance that Pakistan will view enhanced assistance levels in any field as sufficient compensation for abandoning support to these groups". She states bleakly: "The relationship is one of co-dependency, we grudgingly admit – Pakistan knows the US cannot afford to walk away; the US knows Pakistan cannot survive without our support."

Not all the dispatch-writers are that sound. In Georgia, ambassador John F Tefft was assuring his employers only hours before the bombardment of Tskhinvali that nothing of the sort could happen: that was what they wanted to hear. But then we find Margaret Scobey in Cairo, warning Clinton ("Madame Secretary") ahead of her meeting with Egypt's foreign minister that "he may not raise human rights… political reform or democratisation, but you should". Or Tatiana Gfoeller, ambassador in Kyrgyzstan, who reported with amused disgust the ravings of Prince Andrew as he attacked "these (expletive) journalists, especially from the Guardian, who poke their noses everywhere". There's irony there. Those same journalists would print her own secret words and touch off a palace uproar in London.

Britain doesn't cut a pretty figure in the cables. On the rare occasions when US policies – on cluster bomb storage, on rendition flights through UK territory – meet challenges from the UK, British politicians are assumed to be thinking about voters rather than principles. Monotonously, Ambassador Louis Susman in London writes off Gordon Brown's criticisms of Washington policies as posturing "driven by domestic politics".

And the devastating pages about the "special relationship", published in yesterday's Guardian, reveal a trembling British obsequiousness which the Americans find absurd, even embarrassing. Only last year Richard LeBaron, deputy chief of mission in London, said that the British attitude "would often be humorous, if it were not so corrosive". The Tory cringe, as party leaders prepared to take power, is shown to be as low as the Labour cringe when Tony Blair rushed to offer Britain as a so-called "equal partner" in invading Iraq. William Hague, as shadow foreign secretary, assured the embassy in confidence he considered the US his "other country" and promised "a pro-American regime".

This degree of toadying clearly poses problems for the Americans. The dispatches repeat genuine appreciation of Britain's unique loyalty as an ally. But LeBaron was typically shrewd to call this behaviour "corrosive".

The American diplomats are smart enough to know that buttering up the Americans is a routine which incoming British leaders think they have to perform, and that most of them privately resent it. They do it largely for reasons the state department understands only too well. Britain's "independent" nuclear deterrent flies the threadbare rags which are all that remain of the United Kingdom's lost "Great Power" status. But its manufacture and use are in reality dependent on the supply of American technology and American strategic decisions.

But, between the lines, the leaks are telling a bigger, more ominous story. These are exclusively state department documents – not the thoughts of other American power centres with an interest in foreign policy. And these diplomats' reports reveal how far their department has lost prestige and influence. It's a far cry from the days when foreign service giants like Averell Harriman or George Kennan, in the Moscow embassy or in Washington, could issue judgments which would sway a president. Now, though, other agencies – hairier and more shadowy – take it as read that they can require state department officers to carry out their leg work. It's enough to look at the instructions, pretty clearly from the CIA, for US diplomats to spy on their colleagues at the United Nations and even on the secretary-general's office.

Weren't these foreign service men and women humiliated, when they were asked to record the credit card numbers and frequent flyer details of those they worked with? Who asked the embassy in Buenos Aires last year to find out how President Cristina Fernández de Kirchner was "managing her nerves and anxiety", what pills she was taking, and "how does she calm down when distressed"? And "what is the status" of her husband's gastro-intestinal ailment and "what are the most common triggers to [his] anger?" There are spies based in most British embassies, usually with "attaché" cover, but at least MI6 does not order diplomats to collect the intimate personal details of its targets. The professions are kept reasonably separate. So they should be.

It's true that the US system of selecting ambassadors has sometimes been baffling to foreigners. Rich businessmen who donate millions to parties have traditionally been rewarded with embassies (the British, less riskily, reward them with peerages). But these dispatches show that the intellectual quality of the "career diplomat" ambassadors remains pretty high. It would be a disaster for the US if the state department became a "penetrated system" allowing other agencies which, since the Reagan presidency, have progressively pushed state aside to gain the ear of the White House.

Enormous damage was done in the run-up to the Iraq war. As Niall Ferguson puts it in the latest edition of his book Colossus, "responsibility for the postwar occupation of Iraq was seized by the defence department, intoxicated as its principals became in the heat of their blitzkrieg". The state department had laboured hard on long-term plans for the occupation. as the fighting ended. But state had to stand by and see its work junked by Donald Rumsfeld and his neocon team around the Pentagon, who convinced President George W Bush that the Iraqis would simply welcome the Americans as liberators.and romp forward to liberal democracy. The tone of the leaked dispatches suggests thatthis shattering blow to the standing and self-confidence of state has still not been repaired.

Behind all these diligent reports glows an evening landscape, in which a declining empire has lost its way. When communism collapsed, the US expected to become the unchallenged global superpower. But instead the US instantly lost control of countless nations and movements stampeding away from cold war discipline. Paradoxically, it was in those cold war years that America had been in charge of most of the world, mostly by consent, and knew why it was in charge. Now that world has burst into a thousand pieces: all sharp, many of them unstable, some of them fearfully dangerous. And the certainty of mission has gone.

So what is America for in the 21st century? The report-writers are confident about its superior wealth, though it is "banked" by China. They are sure about America's superior military strength, though only a fraction of that strength can be brought to bear in "insurgency" wars. But they are strikingly less sure about America's aims.

In the 1990s the "New American Century" neocons proposed: let's use that wealth and power to act as the world empire we really are! Few traces of that remain. Several ambassadors deny they are playing any great game against Russia or China, because great games are played by empires and the US isn't one. Yet several others indignantly reject the idea of "zones of influence" – no firewall must keep out the benevolent "soft power" influence of America. US policy is stuck aground in muddy places: Israel and Pakistan, Iraq and Afghanistan, Cuba and the Caucasus. If it could extract itself from these, would it simply drift "rudderless" (as the ambassador said about Gordon Brown)?

Perhaps not. Two aims do recur obsessively through these reports. One, rooted in American history, is that the independence of new nations must be honoured and protected. The other is the struggle against nuclear proliferation. Preventing apocalypse has become more important than striving for world leadership. This is a diplomacy clearer about what it doesn't want than what it does.

That's a "mission" we can salute. A British ambassador said: "Our duty at the Foreign Office has been to cover Britain's retreat from greatness and to prevent that retreat turning into a rout." One day the state department may say the same about its service to America.