The coup that transformed the relationship between British politics and journalism began at a quiet Sunday lunch at Chequers, the official country retreat of the prime minister, Margaret Thatcher. She was trailing in the polls, caught in a recession she had inherited, eager for an assured cheerleader at a difficult time. Her guest had an agenda too. He was Rupert Murdoch, eager to secure her help in acquiring control of nearly 40% of the British press.

Both parties got what they wanted.

The fact that they met at all, on 4 January 1981, was vehemently denied for 30 years. Since their lie was revealed, it has been possible to uncover how the greatest extension of monopoly power in modern press history was planned and executed with such furtive brilliance.

All the wretches in the subsequent hacking sagas – the predators in the red-tops, the scavengers and sleaze merchants, the blackmailers and bribers, the liars, the bullies, the cowed politicians and the bent coppers – were but the detritus of a collapse of integrity in British journalism and political life. At the root of the cruelties and extortions exposed in the recent criminal trials at the Old Bailey, was Margaret Thatcher’s reckless engorgement of the media power of her guest that January Sunday. The simple genesis of the hacking outrages is that Murdoch’s News International came to think it was above the law, because it was.

Thatcher achieved much as a radical prime minister confronted by political turmoil and economic torpor. So did Murdoch, in his liberation of British newspapers from war with the pressroom unions, and by wresting away the print unions’ monopoly of access to computer technology. I applauded his achievements, and still do, as I applauded many of Thatcher’s initiatives when I chaired the editorial boards of the Sunday Times (1967-81) and then the Times (1981-2). It is sad that her successes are stained by recent evidence of her readiness to ensure sunshine headlines for herself in the Murdoch press (especially when it was raining), at a heavy cost to the country. She enabled her guest to avoid a reference to the Monopolies and Mergers Commission, even though he already owned the biggest-selling daily newspaper, the Sun, and the biggest selling Sunday newspaper, the News of the World, and was intent on acquiring the biggest-selling quality weekly, the Sunday Times, and its stablemate, the Times.

* * *

Facebook Twitter Pinterest Rupert Murdoch announces his takeover of Times Newspapers in 1981. On the left is the Sunday Times editor Harold Evans and on the right is William Rees-Mogg, editor of the Times. Photograph: Keystone/Getty Images

Times Newspapers had long cherished their independence. In 1966, when the Times was in financial difficulty, the new owner who came to the rescue, Lord Roy Thomson of Fleet, promised to sustain it as an independent non-partisan newspaper – precisely how he had conducted the profitable Sunday Times. Murdoch was able to acquire both publications in 1981 only because he began making solemn pledges that he would maintain the tradition of independence. He broke every one of those promises in the first years. His breach of the undertakings freely made for Times Newspapers was a marked contrast with the independent journalism we at the Sunday Times (and William Rees-Mogg at the Times) had enjoyed under the principled ownership of the Thomson family. Thatcher was a vital force in reviving British competitiveness, but she abetted a concentration of press power that became increasingly arrogant and careless of human dignity in ways that would have appalled her, had she remained in good health long enough to understand what her actions had wrought.

Documents released by the Thatcher Archive Trust, now housed at Churchill College, Cambridge, give the lie to a litany of Murdoch-Thatcher denials about collusion during the bidding for Times Newspapers. They also expose a crucial falsehood in the seventh volume of The History of the Times: The Murdoch Years – the official story of the newspaper from 1981-2002, published in 2005 by the Murdoch-owned HarperCollins. In it Graham Stewart wrote, in all innocence, that Murdoch and Thatcher “had no communication whatsoever during the period in which the Times bid and presumed referral to the Monopolies and Mergers Commission was up for discussion”.

Not just “no communication”, but none “whatsoever”, an overemphasis characteristic of Murdoch in cover-up mode. On the contrary, the Thatcher documents reveal that the extraordinary secret lunch at Chequers in January 1981 specifically focused on his hopes of acquiring Times Newspapers. The file note by Thatcher’s press secretary, Bernard Ingham, is offered as a record of the “salient points” of the meeting, not a full accounting or even minutes. Nonetheless, the note has the aroma of a KGB dead letter drop. In accordance with Thatcher’s wishes, Ingham wrote, he would not let the note go outside No 10. This meant that no one would be told of the fact of the meeting, not even the responsible minister. On Thatcher’s instructions, Ingham’s record of the event was locked away in a Downing Street file marked “Commercial – In Confidence” during her time in office, and later it was not sent to the National Archives. Thatcher was proud of her legal education. She knew she should have refrained from any involvement in what she herself recognised as the “quasi-judicial” procedure for approving a newspaper sale, let alone have afforded privileged access to someone she already knew to have a vested interest. Murdoch’s ambition had been common knowledge since the previous November.

Thatcher knew she should have refrained from any involvement in approving a newspaper sale

Ingham’s “note for the record” bears some parsing. First, the pretence was that Murdoch was afforded a private meeting with the prime minister so she could be briefed on the takeover competition for Times Newspapers. There was no credible reason why Murdoch would be the appropriate person for this task: it was the legal duty of the minister of trade, then John Nott. Second, the prime minister’s “briefing” was from a bidder, who naturally had an urgent interest in rubbishing his competitors. In plain language, this is called cheating. If the prime minister had any shred of justification for intervening, she should have heard from all the competitors. She did not because she was interested only in helping her ally escape the Monopolies Commission. (“He had stood by me in the dark days,” she told an official.) According to the Ingham note, Murdoch, in his briefing, identified only four bidders: Lonrho (a conglomerate led by Tiny Rowland, which would take over the Observer) “unlikely to impress either Thomson or union of journalists”; Robert Maxwell (former MP, fraudster, owner of the Daily Mirror) “a similar verdict”; Sir James Goldsmith (mercurial food tycoon and corporate raider) “probably only for the Sunday Times, a strong contender for that title, though Mr Murdoch felt that anyone willing to buy all the titles was in a stronger position”); and a consortium of Journalists of the Times (“again thought unlikely to carry much conviction, even with the backing of GEC”).

Murdoch apparently chose not to inform the prime minister of other interested and more credible parties. Most conspicuously, it seems that he neglected to mention that the Sunday Times’s management team and the journalists – the very people who had made the paper a towering success – were making their own buyout bid. As Murdoch knew full well, on 31 December 1980, the investment bank Morgan, Grenfell and Co submitted a detailed formal offer on behalf of the group of executives I led as editor and chairman of the Sunday Times executive committee. Nor, it appears, did he mention Vere Harmsworth, the third Lord Rothermere and enterprising proprietor of the Daily Mail. Stewart in his later history of the Times refers to Harmsworth having made a “serious offer”.

Ingham’s record of the meeting looks like a whitewash. Are we to believe that there was no mention at the lunch of the requirement for Murdoch’s bid to be referred to the Monopolies and Mergers Commission? It should have been automatic, given his preponderance of newspaper titles and their circulation in the mass market. He was obsessed with this obstacle between him and his great prize. It was why he asked to meet the prime minister and in secret.

Times Newspapers would have been much on Thatcher’s mind, too, since, whoever prevailed, a change in ownership of Britain’s two leading newspapers would have political consequences. In 1981-2, she was at a low point in her premiership, in the depths of a deepening recession, with the Social Democratic party yapping at her heels on the left, and, on the right, former prime minister Edward Heath, ungrateful at being relieved of the cares of leadership. She had a duty to remind her guest about the legal restraints on media ownership. Decency might have at least caused her to emit a polite cough about their unfortunate obligations. But if she did murmur something, why did Ingham choose not to record it? Ingham is, alas, unable to help us. He has said variously that he has no memory of the meeting and that if there was a meeting it would have been in Downing Street, not Chequers. Somehow or other, on the most politically charged media takeover in British political history, we are faced with an epidemic of forgetfulness. Murdoch could not remember a thing for the Leveson inquiry. Thatcher’s 1993 memoir consigned Murdoch to oblivion. Ingham’s memoir, too, has a large black hole.

On the most politically charged media takeover in political history, we are faced with an epidemic of forgetfulness

Ingham’s note contains a curious record of the exchange at the end of lunch. Had he closed by recording that Thatcher, on saying goodbye, wished Murdoch well in his bid, it would have been polite, if improper in the circumstances. But that is not how Ingham records it. He writes: “The prime minister … did no more than wish him well in his bid …” Why “did no more”? The defensive wording is as suggestive of the core reality of the Chequers lunch as the dog that did not bark in the night in Arthur Conan Doyle’s Sherlock Holmes story Silver Blaze.

Stewart’s history of the Times records the sale transactions as commercial imperatives having no political relevance. John Grigg, Stewart’s predecessor in the Times History series, was less credulous. In this passage from the sixth volume, The Thomson Years, published in 1993, he writes: “Whatever the arguments put forward, the presumption has to be that the decision not to refer was, in reality, political. Though Biffen, as trade minister, was immediately responsible, it was clearly not taken by him alone. The matter was discussed by the cabinet, and the cabinet in Mrs Thatcher’s day was not a notably collegiate body; what she had decided tended to be whatever she wanted it to decide. If Murdoch had shown himself hostile to the prime minister and her party, who can doubt that his purchase of Times Newspapers would have been referred?”

The grossly improper meeting was corroborated by another item in the archives, a handwritten letter to the prime minister from Murdoch’s Eaton Square home. It was to thank her for the lunch that never was. “My dear Prime Minister,” Murdoch wrote, saying he greatly enjoyed seeing her again. Ostensibly, the letter thanked her for letting him interrupt her weekend at Chequers. It is dated 15 January. This is very odd. Murdoch is traditionally punctilious on such matters. The idea just does not scan that Murdoch delayed 11 days to thank his most important connection for the newspaper acquisition of his career. Murdoch is at pains to make a point of emphasising his dilatoriness, first saying his thanks are “belated”, then adding that it is “10 days” since they met. That date of 15 January is an overwritten correction in Murdoch’s hand for another barely discernible date. Nobody knew of the lunch on 4 January, but the later date could be seen as a precautionary way of distancing him from the changes Mrs Thatcher immediately made at the Department of Trade. This is speculation, but things are not what they seem in events suffused with so much subterfuge. Murdoch never moves a pawn without checking he is not exposing a pathway to the queen.

Facebook Twitter Pinterest Rupert Murdoch’s thank you letter to Margaret Thatcher. Photograph: The Guardian

Lord Justice Leveson chose judicial irony as a way of registering disdain for this web of deceit. Of the Thatcher-Murdoch clandestine meeting, he concluded in the final report on his 2011-2012 inquiry into the role of the press and the police in the phone hacking scandal as follows: “That there was a confidential meeting between the then prime minister and Mr Murdoch, the fact of which did not emerge into the public domain for more than 30 years, is troubling in its lack of transparency. It serves as a reminder of the importance of contemporary practice to make public the fact of such meetings. The perceptions at the time and since of collusive arrangements between the prime minister and the preferred bidder are corrosive of public confidence … The prospective deal was plainly of great importance to him. He no doubt believed that there was real value in meeting the prime minister face-to-face, to inform her of his bid and his plans in the event that it was successful, and importantly, to form a personal connection. He would have expected to make a good impression on Baroness Thatcher; he would have known of her respect for risk-taking entrepreneurs and that they would have thought alike on the merits of turning around a troubled newspaper company with industrial relations problems. Their world view had much in common. I have carefully considered what conclusions (whether as to fact or credibility) if any, I should draw from Mr Murdoch’s inability to recall the meeting. It is perhaps a little surprising that he does not remember a visit to a place as memorable as Chequers, in the context of a bid as important as that which he made for Times Newspapers. However, perhaps that is all I need to say.”

Facebook Twitter Pinterest Rupert Murdoch’s thank you letter to Margaret Thatcher. Photograph: The Guardian

* * *

The effect of the secret meeting was immediate. Thatcher went back to London the same Sunday evening to effect a mini-reshuffle that would move responsibility for the Times Newspapers sale from a tough-minded minister to one who, in the words of a cabinet colleague, “had no separate political existence”. She summoned trade secretary John Nott to meet her at 9pm in her flat at No 10. When he went in, he was trade secretary. When he said goodnight, he was on his way to the Ministry of Defence. Nott’s entertaining autobiography Here Today, Gone Tomorrow, describes the scene: “When I got there, I was surprised that [the prime minister’s husband and two of her private secretaries] were also present. The atmosphere was very friendly and informal, and I was offered a drink – about as different from being offered a job by Ted Heath as you imagine. It still seems a little odd to me that it was not a private meeting with the prime minister alone.” Nott was delighted to relinquish Trade. Thatcher immediately telephoned the chief secretary to the treasury, John Biffen, at his home, and offered him Nott’s old position. Her pitch to Biffen was that it would widen his experience. In his diary he wrote that she was “absolutely delighted” when he accepted. For Thatcher, it finished off the day’s work beautifully. January was an unusual month for a mini-reshuffle, giving it something of an emergency flavour. Even so, nobody at the time appreciated the significance of the move, but then nobody knew of the Chequers lunch.

Nott says nothing of it, but his background would have made him a formidable judge of the finances of Times Newspapers and News Corp. He knew how the City operated. Before becoming an MP, he had been the banking pioneer Siegmund Warburg’s protege and became his personal assistant, observing his astute handling of Roy Thomson’s 1966 purchase of the Times. (It is intriguing that when he returned to the City in 1983 as managing director of Lazard Brothers, he helped frustrate Murdoch’s attempt to gain control of the Financial Times.)

John Biffen was a shrewd choice for Thatcher’s objective – and a ruthless one in view of his mental health. The member for Oswestry was popular in the party as a decent and honest old-school Tory, a countryman at heart, but he was also known to be an incorrigible agoniser with a history of mood changes (that would now be more likely classified as bipolar). He was bewildered. Thomson had said they would close the titles unless a suitable buyer came along by March. Biffen’s writings (collated in 2014) reflect the stress he felt: “The episode came at me with great speed. I had no particular knowledge of the newspaper industry … I am certainly encountering the strains of office. I had a restless night and got to the Trade Office laden with anxiety.”

John Biffen was a shrewd choice for Thatcher’s objective – and a ruthless one in view of his mental health

Had the prime minister not been so determined to conceal her meeting with Murdoch, she could have told Biffen that Murdoch saw the Sunday Times as a key to more wealth, more power. Over the Chequers lunch (according to Ingham’s note) she had heard Murdoch enthuse about the commercial prospects of the Sunday Times “… even at the depths of a recession, this newspaper was turning down advertising …” And: “the market clearly permitted an increase in advertising rates”. Murdoch saw the Sunday Times very much as a “going concern”. Thatcher gave Biffen not a hint of any of this. Murdoch was so confident of winning that he had already appointed the man who would run Times Newspapers. This was Gerald Long, the general manager of Reuters. Denis Hamilton, the late chairman of Reuters and also of Times Newspapers, spoke of getting a call from Long on Christmas Day 1980, 10 days before the lunch at Chequers. “He rang to say he was resigning to join Rupert Murdoch’s organisation ... as prospective managing director of Times Newspapers. If, for any reason, the sale of the Times did not go through, however, he wished to keep his job at Reuters. I was flabbergasted.” Hamilton consulted the Times Board. Long was told he must either stay or go. He went, assured by Murdoch it was all pretty well in the bag.

The Monopolies and Mergers Commission was invented to scrutinise proposals and advise the minister of their effect on the public interest. The minister could bypass the commission only when he was satisfied the enterprise was “not economic as a going concern”. It was his decision alone to make and commend to parliament.

Initially, Biffen seemed to recognise that he had no choice but to refer any bid by Murdoch to the commission, since Murdoch already owned the two biggest newspapers in the popular market – and was ready use them to express his own views. In an exploratory meeting Murdoch and his counsel Richard Searby, heard Biffen reaffirm his obligation as secretary of state to make a referral under the act. However, two things stand out: how superficial the scrutiny was and how swiftly the deed was done.

Biffen had barely got a toe under his new desk when he was diverted from the pressing issue of Times Newspapers. He was compelled to leave for India on 16 January on a futile, week-long scheduled trade mission. He was thus far removed from focusing on a dangerous extension of monopoly power that gathered force in his absence. By the time Biffen returned to Heathrow on 25 January, he found his life had become worryingly complicated. “I had hardly left the plane when I was told I would be requested to let the deal go ahead. I was also told Murdoch was the only serious bidder though Maxwell was making noises.” Biffen has not said who was requesting him to let Murdoch through, or who had so egregiously misinformed him on the bidders. He has referred only to “an official”.

The only way a Murdoch takeover of the Sunday Times could be exempt from scrutiny by the Monopolies and Mergers Commission was if Biffen satisfied himself that it was not a “going concern”. Thomson had submitted accounts to the Department of Trade on 19 January when Biffen was in India. By the time he returned, the Sunday Times accounts had been adjusted to show a diminished value.

* * *

The weekend of his return from India, and two days before the Commons had to vote on the takeover, Biffen was in full fret mode. “It’s a nasty problem. All the arguments for a quiet life argue for a reference to the Monopolies Commission. On the other hand, the Murdoch bid had the advantage that it was the bird in the hand and the terms suggested the essential character of the Times would be preserved … I had no strong feelings about Murdoch. How I wished the Sunday Times had been profitable (the Times was too deep in the red) for then the whole deal would automatically go to the Monopolies and Mergers Commission. As it was, the buck was clearly with me.”

John Biffen in 1978. Photograph: David Reed/Corbis

Of course, he had been gravely misled. This was the point where Biffen desperately needed a financial analyst. He did not get one. At the Treasury he had been accustomed to being advised by the sharpest financial brains. Biffen’s permanent secretary, Sir Kenneth Clucas, did not look into the matter at all while Biffen was in India, and said that a meeting on January 26 was his “first personal acquaintance with the issue”. And amazingly, Biffen did not consult the chief financial officer.

On Monday 26 January, a tense Biffen met in his ministerial room with Clucas, officials and his minister of state for consumer affairs, Sally Oppenheim-Barnes (now a baroness). Murdoch was expected shortly after. Biffen’s officials had suggested that on the basis of the new accounts, Times Newspapers was not a going concern, and Biffen records himself as commenting “only in the case of the Times is the issue clear cut”. He recognised that the Sunday Times was in a much stronger financial position than the Times, despite the industrial disruptions that had provoked the sale. The ministers decided to ask Thomson to extend the deadline so that the Monopolies Commission would have eight weeks to vet the takeover if Biffen did decide that way. Then Gordon Brunton, CEO of Thomson London, came into the room to say that a delay would not be workable: it would cut across Thomson’s schedule for ending the interim deals with the print unions.

Murdoch followed Brunton, who was knighted in 1985. “We allotted an hour to seeing him but it was all over in 10 minutes,” said Oppenheim-Barnes. “We had seen the unions before him, who were naturally very concerned. Murdoch came in, sat down and said, ‘Before you begin, I hear you are considering a reference to the Monopolies Commission. If you do that and it ends in allowing me to go ahead, it will mean a long delay. If I don’t get an answer now, I will not pay a penny in redundancy.’ His attitude was, ‘I can’t waste any more time with this.’ He took our breath away. John and I looked at each other and we had no difficulty in reaching a decision.”

Biffen’s hesitation on the viability of the Sunday Times was not reported to the economics committee of the cabinet, summoned in haste and chaired by Thatcher. The minutes record that some members expressed misgivings about the Thomson timetable. Had Biffen’s equivocation with his officials been known, the unease on the left of the Tory party and in the Labour opposition would have been stronger in the next day’s parliamentary debate. Biffen could not bring himself to tell Murdoch and Searby that they would, after all, be spared a monopolies reference. He left it to Clucas, who later said that although Murdoch and Searby received the news decorously enough, he had found them waiting for the lift, rubbing their hands with glee.

Biffen had difficulty writing his account of the cabinet committee. In one passage in his autobiography, he wrote that the PM had been strongly against a referral, but later he was at pains to record that “the prime minister did not make her views known to me directly but I judged she wanted the deal to go forward as I did”. He quieted his doubts about letting Murdoch secure his monopoly by having Clucas look at the restraints on press owners that were created by the vetting committee I served on. Biffen genuinely believed those safeguards against Murdoch would be effective. He was not the first or last to be disabused on that score.

Thatcher pretended throughout that she had nothing to do with ushering the deal through. She told the E committee the decision had been made by Biffen in his “quasi-judicial position”. But she left nothing to chance. She clinched the deal she had contrived, with a coup de grace. She imposed an unusual three-line whip on the Commons vote on January 27.

The deputy editor of the Guardian, David McKie, noted the debilitating effect on Biffen: “The whole proceedings seemed horribly suspect and shabby – all the more disappointingly so when the decision came from the mouth of a minister I greatly liked and admired. And when I next saw him, Biffen seemed decently sheepish, even shamefaced. He wasn’t the sort of minister who would simply say, ‘Margaret told me to do it.’ But I still got the firm impression that he would not have done it without her hot breath on his neck.”

Indeed, in his autobiography Biffen displayed a humble and contrite heart. “The Times deal relied on both the Times and Sunday Times being loss-making. There was no doubt about the former but the Sunday Times was a close-run thing.” His anxieties multiplied when he read my analysis of the fancy footwork with the Sunday Times accounts by Thomson’s London management. He wrote: “Sunday 30 Oct 1983: I am much troubled by the Evans Times business. I am not convinced it will go away. As a consequence I feel low and gloomy.”

Monday 31 October, “dominated by the Evans/Times affair. I had been uneasy as I reflected on the situation and my fears were not dissolved when I met the accountants. They wanted to say as little as possible, but I dug in my toes and got a much more specific handling of the items left out of account when preparing the balance sheet for the 1981 debate …”

Tuesday 1 November: “Evans made a radio charge that it was the biggest cover-up since Profumo. I felt tired and careworn from the Evans business.”

After her re-election in 1987, Thatcher dropped Biffen. He was ennobled as Baron Biffen of Tanat, Shropshire, but embittered by her treatment. He came to see her as a “Stalinist … a tigress surrounded by hamsters”.

Successive governments of both parties, scared when they could have been stalwart, have done no better in dealing with Murdoch. Parliament expresses its determination to protect free speech and plurality but it appears that ministers try hard to live up to Murdoch’s classification of politicians as invertebrates.

The script of the pretend-democracy has been: “What does Rupert want? Hurry, give it to him.” The abuse of the law on media would disgrace a banana republic. In 1983, Murdoch followed his capture of the Times and Sunday Times by laying siege to the Financial Times. He clearly reckoned that the government would come up with another wheeze to spare him a reference to the Monopolies Commission. (It didn’t.) As Paul Carlucci, his man at News America, and later publisher of the New York Post, said: “I work for a man who wants it all, and doesn’t understand anybody telling him he can’t have it all.”

Follow the Long Read on Twitter: @gdnlongread

• This article was amended on 28 April 2015. An earlier version misnamed a former deputy editor of the Guardian, David McKie, as Brian Mackie.

Adapted from the preface to the new fifth edition of Good Times, Bad Times (Bedford Square books and, in the US, Open Road Media). © Harold Evans Associates.

Harold Evans is editor at large for Thomson Reuters.