Highly embarrassing colonial-era files detailing the British army's repressive tactics against Mau Mau insurgents in Kenya during the 1950s will be revealed in a landmark compensation case.

The discovery of thousands of documents withheld for decades from the Kenyan government will raise awkward questions about the Foreign Office's attempt to deny liability for the allegedly systematic mistreatment of thousands of Kikuyu victims prior to independence.

The case, brought by four survivors of the notorious detention camps operated by the colonial authorities could also set a precedent by forcing the release of files relating to other colonies once controlled by the UK.

During the so-called "Emergency", detainees were subjected to arbitrary killings, castrations, sexual abuse, forced labour, starvation and violence from camp guards, lawyers for the detainees will argue in the high court on Thursday.

Among those detained and abused was Barack Obama's grandfather.

The UK government denies that mistreatment was as widespread as alleged. In the past it has relied on an obscure legal precedent relating to Patagonian toothfish which states that responsibility for acts committed by a colonial government pass to the new, successor government at independence.

An open letter from the Kenyan Human Rights Commission to the foreign secretary has been signed by a host of civil rights organisations and politicians including Archbishop Desmond Tutu, Vince Cable, and Professor Sir Nigel Rodley, a former UN special rapporteur on torture.

It declares: "It is a little known fact that during the Mau Mau war, in the run up to Kenyan independence, the then British government systematically violated human rights and committed war crimes on a vast scale.

"The full extent of Britain's knowledge and authorisation of torture only emerged recently after historians from Harvard and Oxford gathered detailed testimonial evidence and conducted extensive research of public and court records.

"In response, the government claims that the current Kenyan government is legally liable for abuses which took place under the British colonial administration. In our view, this represents an intolerable abdication of responsibility."

Key documents uncovered during research into the claims show that the Foreign and Commonwealth Office made a calculated decision not to hand over any of its colonial era files to the Kenyan government.

A letter dated 7 November 1967 stated that it was general practice at independence not to hand over any files that "might embarrass HMG or other governments" or "members of the police, military forces, [or] public servants".

The FCO letter continued: " … The moment we return any records whatsoever there is the danger that we should find ourselves under constant pressure to make good other gaps … in the record of the Kenya government.

"… The fact that it has always been British policy to withdraw or destroy certain sensitive records prior to independence has never been advertised …"

The Kenyan government at the time was requesting the return of files relating to the Mau-Mau Emergency and other topics.

The Foreign Office will have to argue that the Kenyan government is responsible for pre-independence human rights abuses despite the fact that it did not then exist and was subsequently denied access to files revealing what actually occurred during that period.

Dan Leader, of Leigh Day & Co, the law firm representing the Kenyan claimants, said: "Every leading historical expert on the Kenya Emergency has filed statements in support of the victims. To seek to pin the liability for British torture onto the Kenyan government is an appalling stance for the government to take and one which we hope the judge will reject."

An FCO spokesman said: "We understand the strong feelings that the Mau Mau issue still creates in Kenya and elsewhere. The Emergency period caused a great deal of pain for many on all sides, and marred progress towards independence.

"The UK intends to fully defend these cases. The defence will be based on well-established aspects of constitutional and international law. We are stating that under the law HMG cannot be held liable in this case."

Explaining the belated discovery of the FCO-witheld files, Foreign Office minister Lord Howell told parliament his department had "decided to regularise the position of some 2,000 boxes of files it currently holds, mainly from the 1950s and 1960s, which were created by former British administrations overseas. The intention is to make as much of this material as possible available to the wider public.

"The domestic records of colonial administrations did not form part of British public [ie, official] records and they were kept by the individual states created at independence. It was however the general practice for the colonial administration to transfer to the United Kingdom, in accordance with colonial office instructions, shortly before independence, selected documents held by the governor which were not appropriate to hand on to the successor government."