It is the historian's dream. It is the diplomat's nightmare. Here, for all to see, are the confidences of friends, allies and rivals, garnished with American diplomats' frank, sometimes excoriating assessments of them. Over the next couple of weeks, you, the readers of the Guardian, will enjoy a multi-course banquet from the history of the present.

The historian usually has to wait 20 or 30 years to find such treasures. Here, the most recent dispatches are little more than 30 weeks old. And what a trove this is. It contains more than 250,000 documents. Most of those I have seen, on my dives into a vast ocean, are well over 1,000 words long. If my sample is at all representative, there must be a total at least 250m words – and perhaps up to half a billion. As all archival researchers know, there is a special quality of understanding that comes from exposure to a large body of sources, be it a novelist's letters, a ministry's papers or diplomatic traffic – even though much of the material is routine. With prolonged immersion, you get a deep sense of priorities, character, thought patterns.

Most of this material is medium-and high-level political reporting from around the world, plus instructions from Washington. It is important to remember that we do not have the top categories of secrecy here – Nodis (president, secretary of state, head of mission only), Roger, Exdis, Docklamp (between defence attaches and the defense intelligence agency only). What we have is still a royal banquet.

Small wonder the state department is crying blue murder. Yet, from what I have seen, the professional members of the US foreign service have very little to be ashamed of. Yes, there are echoes of skulduggery at the margins, especially in relation to the conduct of "the war on terror" in the Bush years. Specific questions must be asked and answered. For the most part, however, what we see here is diplomats doing their proper job: finding out what is happening in the places to which they are posted, working to advance their nation's interests and their government's policies.

In fact, my personal opinion of the state department has gone up several notches. In recent years, I have found the American foreign service to be somewhat underwhelming, reach-me-down, dandruffy, especially when compared with other, more confident arms of US government, such as the Pentagon and the treasury. But what we find here is often first rate.

As readers will discover, the man who is now America's top-ranking professional diplomat, William Burns, contributed from Russia a highly entertaining account – almost worthy of Evelyn Waugh – of a wild Dagestani wedding attended by the gangsterish president of Chechnya, who danced clumsily "with his gold-plated automatic stuck down the back of his jeans".

Burns's analyses of Russian politics are astute. So are his colleagues' reports from Berlin, Paris and London. In a 2008 dispatch from Berlin, the then grand coalition government of Christian and Social democrats in Germany is compared to "the proverbial couple that hated each other but stay together for the sake of the children". From Paris, there is a hilarious pen portrait of the antics of Nicolas (and Carla) Sarkozy. And we the British would do well to take a look at our neurotic obsession with our so-called "special relationship" with Washington, as it appears in the unsentimental mirror of confidential dispatches from the US embassy in London.

Reassuringly, we also find occasional signs of the British Foreign Office standing up for our values. According to a report from 2008, one senior British diplomat, Mariot Leslie, "was very frank that HMG did object to some of what the USG [government] does (eg, renditions) and therefore does have some redlines".

It is very disturbing to find telegrams signed off by Hillary Clinton which seem to suggest that regular American diplomats are being asked to do stuff you would normally expect of low-level spooks – such as grubbing around for top UN officials' credit card and biometric details. Clarification is now urgently needed from Foggy Bottom (the seat of the state department) of who exactly was expected to do what under these human intelligence directives.

More broadly, what you see in this diplomatic traffic is how security and counter-terrorism concerns have pervaded every aspect of American foreign policy. But you also see how serious the threats are, and how little the west is in control of them. There is devastating stuff here about the Iranian nuclear programme and the extent not merely of Israeli but Arab fears of it ("cut off the head of the snake", a Saudi ambassador reports his king urging the Americans); the vulnerability of Pakistan's nuclear stockpile to rogue Islamists; anarchy and corruption on a massive scale in Afghanistan; al-Qaida in Yemen; and tales of the power of the Russian mafia gangs, that make John le Carré's latest novel look almost understated.

There is a genuine public interest in knowing these things. The Guardian, like the New York Times and other responsible news media, has tried to ensure that nothing we publish puts anyone at risk. We should all demand of WikiLeaks that it does the same.

Yet one question remains. How can diplomacy be conducted under these conditions? A state department spokesman is surely right to say that the revelations are "going to create tension in relationships between our diplomats and our friends around the world". The conduct of government is already hampered by fear of leaks. An academic friend of mine who worked in the state department under Condoleezza Rice told me that he had once suggested writing a memo posing fundamental questions about US policy in Iraq. "Don't even think of it," he was warned – because it would be sure to appear in the next day's New York Times.

There is a public interest in understanding how the world works and what is done in our name. There is a public interest in the confidential conduct of foreign policy. The two public interests conflict.

One thing I'd bet on, though: the US government must surely be ruing, and urgently reviewing, its weird decision to place a whole library of recent diplomatic correspondence on to a computer system so brilliantly secure that a 22-year-old could download it on to a Lady Gaga CD. Gaga, or what?