Historic papers about the slave trade are among the enormous cache of public documents that the Foreign Office has unlawfully hoarded in a secret archive, the Guardian has learned.

Some of the papers appear to date back to 1662 and are thought to contain information about England's involvement in slavery, while others were created in the 19th century and detail British attempts to suppress the trade.

They are contained in a vast archive of 1.2 million files that the Foreign Office has kept at Hanslope Park, a high-security compound that it shares with MI5 and MI6 in the Buckinghamshire countryside north of London.

Under the Public Records Acts, the slavery papers should have been handed over to the National Archives in Kew, south-west London. The Foreign Office has not answered questions about the papers, and historians say it is difficult to be sure of their significance without having an opportunity to examine them.

William St Clair, senior research fellow at London University and author of door of The Door of No Return, a study of the Atlantic slave trade, said he doubted that the hidden papers would "cause embarrassment or difficulty today" but raised the possibility that they might provide background information for people seeking financial compensation for the manner in which their ancestors were treated.Nick Draper, who helps to run the Legacies of British Slave-ownership research project at University College London said he too doubted that any of the documents were likely to trigger a wholesale revision of the history of Britain and slavery, but wondered whether they detailed the continuing involvement of British merchants after the trade was abolished in the British empire in 1807.

St Clair – who was a senior civil servant at the Foreign Office before becoming a historian – added that in the absence of a written constitution, assorted statutes such as the Public Records Acts "are essential components of democracy and public accountability", and that the department was in clear breach of the law.

A number of the country's leading historians have expressed anger and alarm at the way in which huge numbers of files have been unlawfully hidden from public view. Several are considering whether legal action may be needed in order to preserve the Hanslope Park archive and to secure access, and some have questioned whether their major works on such subjects as the outbreak of the first world war may need to be rewritten.

The scale of the hidden archive is demonstrated by an inventory that the Foreign Office has published, which appears to show that one of the listed items may itself contain 2.9 million documents.understood to occupy around 15 miles of shelving at Hanslope Park, although this is difficult to verify as the Foreign Office says it will not permit members of the media to enter the compound.

The inventory shows that it contains an enormous amount of documents about Hong Kong and Rhodesia. There are files about the UK's claim on the Falkland Islands and the war of 1982; German-language documents about the Holocaust that may have been removed from postwar Germany; and a number of files about the activities in Northern Ireland during the early 70s of the Foreign Office cold war-era propaganda unit that was known as the information research department.

Hanslope Park in Buckinghamshire, where the files are held. Photograph: Martin Argles for the Guardian

One file has the title Australian War Crimes, while another is entitled Nazi War Criminals in Britain. There is a file about the disgraced publishing tycoon Robert Maxwell. One batch of documents is said to contain "communications with General Gordon; fall of Khartoum".The existence of the archive came to light as a result of proceedings in the high court during which a group of Kenyan pensioners successfully sued the British government for compensation for the abuses they suffered while detained during the 1950s Mau Mau insurgency.

After repeatedly assuring the court that it had disclosed all its historic documents, the Foreign Office admitted it was holding 1,500 Kenyan files at Hanslope Park. That admission was made only after the department was served with a witness statement from a historian who had located a 45-year-old Whitehall memo that referred to the material.

Ministers then informed parliament that there were a total of 8,800 files from 37 former colonies being stored at Hanslope Park; when these were finally handed over to the National Archives at Kew, the true figure was found to be nearer 20,000.

What the Foreign Office did not disclose at that time was that the colonial-era files were just a tiny part of the vast repository at Hanslope Park. Instead, it has since acknowledged, it asked the justice secretary, Chris Grayling, to sign an authorisation for the retention of 1.2 million files, putting them on a legal footing for the first time while a plan could be devised for their transfer to Kew. That was done without any public announcement.The exact number of files within the archive that have been withheld in breach of the Public Records Act is unclear. Initially, the Foreign Office said there were 1.2 million. Earlier this year, this figure was revised downwards to 600,000, with officials maintaining that the remaining 600,000 were not yet due for release under the 30-year rule. However, the department's own inventory shows some of those files date back to 1852.

Similarly, the department is now claiming that the part of the archive that it does not deny has been held unlawfully had only recently "come to light", yet the inventory shows that it includes two boxes of files described as Chilcott [sic] inquiry papers dated from 2001 to 2009, and six boxes of papers concerning investigations by the Serious Fraud Office.

The SFO documents include a number created in 2010 during the inquiry into acts of bribery by British arms company BAE Systems.

The Foreign Office has presented its plans for the release of some of the Hanslope Park files during a meeting of the National Archives' advisory council, which usually scrutinises government departments' requests to retain or redact a small number of files beyond the 30-year disclosure rule. The meeting, held last November, was closed to the public.

In a statement to MPs the following month, Foreign Office minister David Lidington said a portion of the files would be transferred over a six-year period.

However, it remains unclear what proportion of the archive will be transferred during this period. Although Lidington said the Foreign Office was "committed to meeting our public records obligations in as transparent a manner as possible", the department has released no details of its transfer plan, declined to say how long it will be before all the files are made public and given no details of expected cost.

• This article was amended on 22 January 2014. One of the paragraphs had become incomprehensible in the editing process, leading to a key quote from William St Clair being omitted. This has now been corrected.