The remit of the US Federal Bureau of Investigation (FBI) includes monitoring figures who potentially threaten national security. And the FBI has long included famous writers on that list.

To them, writers pose a double menace: not only do they pose a potential threat themselves, they might also inspire large groups of people to undermine the status quo, which the FBI is charged with protecting. The perceived threat posed by novelists and essayists is laid bare in Writers Under Surveillance: The FBI Files, a new book comprising facsimiles of archived files on famous American authors. During the Cold War, when suspicion of writers and intellectuals was at its peak, the FBI was under the control of the domineering, aggressive and thin-skinned J Edgar Hoover. During his long career as FBI director, Hoover took an active personal interest in pursuing political and personal opponents. Writers Under Surveillance reveals all this in official memoranda, letters and reports, with redactions, mainly to conceal identities of informants and other intelligence agencies. The editors have selected the more complete documents.

Subjects covered in this book include counterculture writers Ken Kesey (inter-state fugitive), Aldous Huxley (promoting hallucinogens), Norman Mailer (supporting Communist Cuba), and Susan Sontag (visiting Communist Hanoi contrary to standard US passport interdictions). Beyond committing an actual crime, various behaviours sparked FBI attention: Communist sympathy, threatening government officials, defecting from Communist countries, agitating for drug legalisation, travelling to Cuba, homosexuality, and criticism of Hoover or the FBI.

For example, James Baldwin was involved in activism supporting the civil-rights movement and was associated with the Nation of Islam. He gave interviews to foreign newspapers that showed American society in a state of strife under an authoritarian government. When Baldwin’s activities were brought to the attention of the Hoover, he asked: ‘Isn’t Baldwin a well-known pervert?’ Baldwin was homosexual, but that was not publicly known. Gay people were vulnerable to blackmailing by the FBI, as well as agents of foreign governments. The FBI debated blackmailing or smearing screenwriter Terry Southern over his extensive collection of pornography. The documents here include letters to the FBI tipping it off about seditious writers. Often these were the result of little more than personal grudges against individuals. Ayn Rand was an ardent anti-Communist and volunteered as an informant. She had seen an interview with Hoover in which he described himself as an ‘objectivist’, ie politically neutral, which she enthusiastically took to mean that he was an adherent of her political philosophy of objectivism. Hoover wrote on her letter, ‘I have never said I was an “objectivist” – whatever that is’.

Memoranda from 1942 concerning Ernest Hemingway discuss his maritime surveillance work for the US embassy in Havana under the guise of marlin fishing. For $500 a month, Hemingway paid a team of 26 sailors, barmen and waiters as informants gathering information on German U-boat activity. Hemingway later claimed that he made the arrangement primarily in order to get fuel for his boat, but reports here suggest he approached the work seriously (if haphazardly). Most of Hemingway’s tips were erroneous. An agent impatiently concluded, ‘the bureau has by careful and impartial investigation from time to time disproved practically all of the so-called Hemingway information. I don’t care what his contacts are or what his background is – I see no reason why we should make any effort to avoid exposing him for the phoney that he is.’ During Hemingway’s last period of encroaching dementia, his claims that he was under surveillance were seen as signs of paranoia. As it happens, he was being monitored, just less actively than he imagined.

Frequently, the documents tell us less about the subjects than they do about the agents and informants. One can imagine the faces of agents reading a report about Beat poet Allen Ginsberg turning up at a protest with his lover. ‘[They] carried little Japanese finger cymbals and chanted Hindu prayer formulas directed to Shiva, one of the Hindu principle [sic] deities. [They carried] a sign reading “Smoke pot, it is cheaper and healthier than liquor”.’ Browsers of this book can also find a copy of Hunter S Thompson’s driving licence and the security-clearance check for Tom Clancy’s appointment as a national-security adviser. (None of his neighbours had a bad word to say about him.) The FBI received a letter from a student wanting information about the recently arrested Ken Kesey for a term paper. ‘Please rush this information if possible. Paper is due in two weeks.’

When I started this book, I expected to feel chilled by secret-service intrusiveness. Yet by the time I had finished it, what I had encountered seemed positively quaint. Compared to the memoranda and surveillance reports published here, today hundreds of times more data on ordinary people is already in the databases of Google, Apple and Facebook – data so intimate, comprehensive and precise that entire lives can be reconstructed through financial transactions, internet browsing history, geo-located photographs, phone records, CCTV footage, smart travel-cards and dozens of other data trails we have little or no chance to control or even monitor. Governments, corporations and hackers know vastly more about ordinary citizens than the world’s best security services ever knew about high-profile dissidents. That’s far more chilling than even the worst McCarthyite-era examples of surveillance in this book.