American exceptionalism kicked off with a defining statement from John Winthrop, the first governor of the then Massachusetts Bay Colony in the early decades of the 17th century. Surveying the wilderness in front of him – the eastern edge of a continent that was the ultimate terra incognita – Winthrop vaingloriously noted: “We shall be as a city upon a hill, the eyes of all people are upon us.”

How ironic that the most platitudinous of recent presidents, Ronald Reagan, chose to cite Winthrop’s commentary in one of his most oft-quoted speeches. Did Reagan’s preferred scribe, Peggy Noonan, realise she was borrowing a statement from the head of a puritanical hierarchy; a group that thought nothing about clapping non-believers in the stocks, hanging heretics for preaching the benign tenets of Quakerism, burning witches, and running its highly theocratic colony in a manner not dissimilar from the Taliban several centuries later? The Grand Guignol grotesqueries of religious fanaticism are indeed timeless.

No wonder that those men of the enlightenment who wrote the American Constitution – Jefferson, Adams, Madison, Franklin – were so insistent on the separation of church and state. Because they knew of the pious monomania (and the tendency toward mob zealotry) that were ingrained from the start within the national body politic. No wonder that the first truly landmark American novel, Nathaniel Hawthorne’s The Scarlet Letter (1850), examined the public shaming of a New England innocent, Hester Prynne, forced to wear the letter A around her neck after being falsely accused of adultery in Puritan-era Massachusetts. No wonder that, 100 years after Hawthorne published his daring riposte to that very American need for communal moral self-righteousness, the absolute evil of the McCarthyite witch hunt commenced. Spearheaded by the deeply dipsomaniac and demagogic Senator Joseph McCarthy, it insisted on its victims wearing the metaphoric equivalent of a red C (for Commie) around their neck.

More telling was the denunciation system he devised, which was ethically so pernicious. If the alleged leftist did not name names and reveal their one-time fellow travellers, he or she would be placed on a blacklist and rendered unemployable. And the victim knew that, besides committing an act of professional suicide, such moral heroism would count for nought among those executives in Hollywood or New York who controlled the cultural landscape.

Of course, McCarthy overstepped the moral mark and ended up in disgrace, dying from massive cirrhosis of the liver shortly after his scorched earth crusade against the Red Scare was exposed as a total sham. But though there was much communal hand-wringing in the wake of the blacklist, political concern about artists as subversives continued under more clandestine circumstances, especially given that the man wielding the most power within the American secret state, J Edgar Hoover – the infamous director of the FBI for more than 36 very long years – ran this national intelligence organisation as his private fiefdom.

Hoover was, as the poet Theodore Roethke noted, “the head of our thought police – a martinet, a preposterous figure”. He was also a virulent anti-Communist who relished exposing the sexual inclinations of those under his investigation. He was especially preoccupied with gay men who expressed allegedly subversive views (ie writers). Then again, he himself was a man with just a few private contradictions: Mr Oedipus Complex who lived with his mother until her death (when he was well into middle age), who was rumoured to enjoy cross-dressing and had a male consort who might just have been his lover. And, of course, he was someone who deeply subscribed to the theory (popular among arch-patriotic Yahoos) that there was a direct link between communism and homosexuality… thus leading to the following variation on a joke I heard years back while working in the theatre in New York: “Is J Edgar Hoover gay? Oh no, no, not at all. But his boyfriend is gay.”

Hoover never feared public exposure of his alleged gayness, at a time when such a revelation was a career-ending event, because he had so much dirt on everyone else and was so deeply dreaded. But he no doubt relished his agents’ reports on writers such as James Baldwin: “On the subject of homosexuality Baldwin stated ‘American males are the only people I’ve ever encountered in the world who are willing to go on the needle before they go to bed with each other. Because they’re afraid of this, they don’t know how to go to bed with women either.’ ”

Allen Ginsberg – another openly gay man in an era when that was an act of defiant courage – won big black marks from the FBI not just for his sexuality or a visit to Cuba in 1965, but also for his “dangerous” pro-weed views. Just consider this FBI surveillance report on Ginsberg and his lover Peter Orlovsky at a legalise marijuana rally in Manhattan’s East Village: “[they] carried little Japanese finger cymbals and chanted Hindu prayer formulas directed to Shiva… GINSBERG described Shiva as the god of meditation, yoga and meditation. GINSBERG predicted that marijuana will be recognised in the United States within five years.”

It took another 40 or so years, but weed is legal in many places now stateside. Ginsberg got it right (so too, I suppose, did Shiva).

Naturally, the hyper-macho Ernest Hemingway ran afoul of the Feds, for referring to the FBI’s legal attaché in pre-revolutionary Cuba as “a member of the American Gestapo”. Hoover seemed very au courant with Hemingway’s declining mental health, culminating in his gunshot suicide in 1961, and also had Terry Southern, author of a hugely best-selling pornographic novel of the Sixties, Candy, investigated for possible prosecution. After all, free love, in the FBI’s world view, was dangerous to national stability. However, as the agent playing literary critic reported to J Edgar: “The reviewers, while not uniformly praising Candy, have generally recognised it as an amusing and effective lampoon of ‘dirty’ books.” As such, he did not recommend prosecution.

Reading through dossier after dossier on 16 American writers contained in Writers Under Surveillance: The FBI Files, what strikes you immediately is the terrifying absurdity of Hoover’s obsession with anyone who didn’t follow his patriotic party line and dared to express critical concern about the national psyche in well-written words. Susan Sontag got a big file opened on her for visiting Hanoi during the Vietnam War and speaking out on the bellicose campaign of “American intervention”. The African-American sociologist and civil rights advocate, WEB Du Bois, was investigated for being a communist (even though he was known not to be) because he was “in sympathy with the Southern Negro Youth Congress”. The FBI agent did mitigate this with the following racist comment – that Du Bois was considered “one of the most outstanding and competent negroes in Atlanta”.

As can be gathered, this remarkable volume makes for compulsive and deeply unsettling reading. Compiled by MuckRock, a laudable and important “non-profit collaborative news site”, its editors have used the still viable and crucial US Freedom of Information Act (FOIA) to obtain the manifold files which show that even a right-wing true believer such as the infamous Ayn Rand didn’t escape Hoover’s concern – over her “vocal atheism”. It is an understatement to say that there was something decidedly Theatre of the Absurd on the FBI’s part in fearing that Truman Capote was a threat to national security for signing a “Fair Play for Cuba” petition (especially given his beau monde predilections).

As you peruse hand-typed classified document after document – frequently with key passages blacked out by the secret powers that be before they were released through the FOIA – you cannot help but marvel at the paranoid insanity of all secret state organisations. Just as the East German Stasi’s aim was “to know everything about everyone”, Hoover and his equally compulsive operatives believed that anyone who raised a questioning voice about the blessed American Way of Life was worthy of serious scrutiny.

This was especially true of writers for their ability to think independently; to view life in non-Manichean terms; to expose national hypocrisies; to question the conformist status quo; and to see that calling to account our immense internal contradictions is patriotism in the best sense of the world – one free of the xenophobia, the toxic nationalism, the “you are with us or against us” message that Hoover pioneered and which is a key construct of the Trumpian world view.

Hoover died in 1972. But as this essential volume makes known, the surveillance of American writers hardly died with the man Nixon referred to as ‘“that old cocksucker”. And I suppose there is something darkly reassuring that, in this increasingly post-literate age of ours, the thought police still consider writers to be dangerously influential. Only now, their surveillance is carried out courtesy of that smartphone in your pocket.

Douglas Kennedy’s new novel, “The Great Wide Open”, is published by Hutchinson this month

Writers Under Surveillance: The FBI Files

Edited by JPat Brown, BCD Lipton, Michael Morisy

The MIT Press, 400pp, £20