I was two years old when my activities first came to the attention of MI5. In 1952, nothing if not thorough, a security service officer carefully filed a copy of an article written by my father in the Daily Worker about the books good Communists should encourage their children to read. The answer, my father wrote, was complicated by the age factor. "My son (aged 2½) adores Tootles the Train [see footnote] but would scarcely enjoy Kidnapped yet."

And there the truth about Tootles and me might have remained. But someone in MI5 decided last year it was time to place 12 files of surveillance records on my father, covering the years 1938 to 1960, in the public domain. Why his files were released, I have no way of knowing. But they are in the National Archives and I have read them. They reveal lots about my parents. But they also say lots that is freshly topical this week about the logic and limits of attempts to monitor political threats.

It was never a secret that my father, Arnold Kettle, was a Communist. He joined the Communist party in 1936 as an undergraduate at Cambridge and he was still a member when he died 50 years later. He spent most of his life as a university English teacher, wrote Marxist books on the novel, and was on the CP's inner executive committee. All this was public knowledge. But my father was also a member of the 1930s Cambridge Apostles, along with the spies Anthony Blunt and Guy Burgess, both of whom he knew, and he made occasional visits behind the Iron Curtain after the war. Was there, perhaps, more to his political activites than he ever admitted? Was he even in some way a spy?

Arnold was certainly a secretive man. And a disciplined one. His loyalty to the Communist party rarely wavered. Only after he died, for example, did I discover that, as a member of the CP executive in 1956, he had voted in a minority of two to condemn the Soviet invasion of Hungary. But the vote went against him and, as he believed in party discipline, he never referred to it again.

But the most striking aspect of the MI5 records on him is paradoxical. They are simultaneously thorough and inadequate, a point echoed in recent criticisms of the service today. Arnold's files are full of carbon copies of innumerable memos and reports. But he was a medium-sized target in a huge operation. By 1952, according to Christopher Andrew's history, MI5 knew the names of 90% of the CP's 35,000 members — not least by bugging the party's Covent Garden headquarters, but above all because they had acquired the party membership lists.

My father first crops up in MI5's files in 1938 in what now seem romantic circumstances, a member of a student delegation to Barcelona during the Spanish civil war. During the second world war, he applies to work in intelligence, but is rejected as "highly undesirable". When he goes for officer training , MI5 keeps in touch with his officers. "I have observed no evidence of anything subversive in his conduct ... Kettle is not a striking figure physically," one of them writes back.

After the war, his marriage to my mother is noted – "she is an ardent communist and seems to be present when anything of note is taking place in the left-wing world in London" – and they move to Leeds. His mail is opened, and there are phone intercepts whenever he contacts the CP headquarters or stays with prominent Communists during London visits. His bank accounts are monitored. In 1952, with Doris Lessing and others, he visits the Soviet Union. An MI6 report describes him as "an intellectual with a clever approach to communism when talking to the unconverted, in that he appears to ask searching or suspicious questions concerning the regime in the USSR but sees that the answers are always favourable to the Soviet side". That gets it about right.

Sometimes the reports widen into less conventional subjects. Cambridge police report to MI5 about a meeting "at which Dr Arnold Kettle will speak on Hamlet". An intercepted letter, carefully filed for posterity, asks for Arnold's views on Shelley. A note reveals that Arnold has defended EP Thompson's views on William Morris in a party meeting. A party official asks why Arnold's new book is "imbued with the Leavis critical approach" and contains "nothing approaching Marxist criticism". In 1958, Arnold writes a letter to the party urging them not to denounce Boris Pasternak's Dr Zhivago, "a work of great genius by a man of extraordinary intelligence and honesty". John Berger writes back, intercepted again, to congratulate him.

Occasionally, there is something more serious. An invitation from party HQ to take part in a meeting on "the development of automation and automatic devices which are substituting larger numbers of clerical and accounting workers" leads to him being tailed across London by the legendary Jim Skardon, interrogator of Kim Philby. Then, during a 1958 visit to Leeds, the Communist party leader John Gollan, who is being tailed, gives him a box, thought by MI5 to contain money, which he places in his bank for a year, before giving it back to Gollan. It looks as though Moscow gold passed through our house. Certainly many Indian and African students did so; "they all appeared to be intellectual types", a watcher reported.

On a personal level, these 12 files are of course hugely fascinating to me. But what wider lessons, if any, do they contain? I think MI5 was right to try to monitor the Communists. At least in its early days, the CP wanted to overthrow capitalism and transform the British state – and it was being financed by a hostile power to do so. MI5 could hardly look in the other direction, even when it was clear, as it certainly was by the 1950s, that the CP wasn't going anywhere as a revolutionary force and was increasingly looking for democratic and liberal legitimacy. My father was a small part of that, not a spy, but some monitoring of our lives made sense.

Maybe there are more damning files elsewhere. In their absence, though, the picture in these documents is of someone who was not so much wicked or threatening as wrong. "Nobody any longer believes socialism won't work," Arnold is reported to MI5 as saying in 1959. Half a century on, plenty of people believe the opposite. My father got a lot of the individual issues – Spain, Hungary, Stalin, Pasternak and certainly Hamlet – right. But he and his comrades got the big call very wrong indeed. Reading these files does not make me shocked so much as sad.

Ironically, my father did have a real secret. His family have known it for years. Doris Lessing wrote about it in her autobiography. And MI5 knew about it too. "It has been suggested from a somewhat doubtful source that Dr Kettle may have homosexual tendencies," they noted in 1953. The source was right. Five years later, the watchers reported: "Kettle attends meetings of the political committee in London on the second Saturday in each month, following which he makes contact with homosexual friends in St Martin's Lane and Soho."

Half a century on, much of the technology of surveillance has changed out of recognition from those distant days. So have the threats. But the pressure to monitor the potentially dangerous is just as great as it was in the cold war era. I want MI5 to protect us from bombers like the 7/7 jihadis or butchers like the Norwegian racist. I'd like to think they have become better at sorting out the lethal from the harmless and the public from the private. "It is a wonder that you can pick as many locks leading into the hearts of the wicked men as you do," Oliver Cromwell's son wrote to England's spymaster John Thurloe. Then as now, the effort comes at a high price.

• This footnote was appended on 29 July 2011: Tootle, the story of a young locomotive, by Gertrude Crampton was published in 1945. Tootles the Taxi by Joyce B Clegg was first published in 1956 with a revised edition by Lynne Bradbury published in 1984.