When you lock people up without trial, “whether it is in Belmarsh or Guantánamo Bay, you recruit more terrorists than you contain”, warned Shami Chakrabarti earlier this year. She has said that senior intelligence officials made a successful bid for enormous power post-9/11, particularly in government surveillance. And appalling attacks in France and Belgium have made us more likely to give the security services the power to curtail our liberties. But should we?

I claim some special insight. For at least the first 10 years of my life, from 1945 to 1955, my childhood home was under constant MI5 surveillance. My parents knew it, though I did not; and I have spent a lot of the past two years with recently released MI5 records at the National Archives, reading verbatim my parents’ telephone conversations from 60 years ago and the spies’ reports on their movements.

I recognise a few of the people described as the adults who peopled my childhood, and I can picture in my mind where the watcher’s car was parked when he reported on my family’s movements. My father, John Beckett, was a leading fascist. He was in prison for nearly four years during the second world war, then under a sort of house arrest, not allowed to live within 20 miles of London or to travel more than five miles from his home.

When these restrictions were lifted, the surveillance continued, masterminded by Graham Mitchell, who became deputy director of MI5 in 1956. It is mostly Mitchell’s memoranda I have been reading. They betray a deeply unhealthy pleasure in the secret power his position gave him over the life of another man. At one point Mitchell believes (probably wrongly) that he has caught John out in an extramarital affair. Telephone intercepts have John telling his wife he is having dinner with a male friend, but the man following him says it is a female friend who joins him in the restaurant.

Mitchell was also slyly amused to have convinced John that one of his friends was, in fact, an MI5 spy. “Mr Mitchell”, reads an internal memorandum. “You may be amused to know that John Beckett is certain that Major Edmonds gave information to MI5 … Beckett believed that Edmonds tried to seduce his wife while Beckett was in prison.”

What was true of fascists in the 1940s and 50s is true of suspected Islamic extremists in 2016

Just after the war my father, against all the odds, managed to get himself a job as assistant administrator at a local hospital. When he was fired, a few months later, he thought it was because he had bought bananas for a desperately sick baby on the black market. But the real reason was that Mitchell seems to have had a quiet word. My mother had always hoped her husband would settle down to a quiet, normal job and give up politics. But when he lost the job, he returned to the only work he could get, running a neo-fascist party called the British People’s party and being paid to do so by its patron, the Duke of Bedford.

From all I have read, I draw two conclusions.

First, Chakrabarti is right. My father came out of prison far more racist – and, in particular, antisemitic – than he went in: a phenomenon familiar to those who have studied wartime detention.

After the war, the constant surveillance, which he knew was there but could never pin down, made him just a little mad. He was noisy and entertaining, he could tell a good anecdote, but there was something strange about him. And sometimes he would say something about a race – about Jews or about black people – so gross and offensive that, even as a child in the 1950s, it made me start and stare.

John was proud of holding himself together in prison when others went to pieces, but this came at the cost of internalising his rage. This, I think, stayed with him during the years after the war, when he knew they were watching him.

Second, such a surveillance regime is very bad for the characters of those who govern us. Reading Mitchell’s memoranda, the satisfaction in the covert control he exerted over other people shines through his flat prose.

If I had been home secretary, I too would have locked up the Oswald Mosleys, the John Becketts and the rest of them

Yet we must be protected. And in May 1940 we had to be protected against people like my father. That was one of the few moments in history – with the Germans apparently just a step away from an invasion – when internment without trial was justified; and if I had been home secretary, I too would have locked up the Oswald Mosleys, the John Becketts and the rest of them.

But by 1942 the fifth column had been shown to be a chimera, and an independent panel set up to oversee internment had advised release. Mitchell’s memoranda and those of his political boss, the home secretary Herbert Morrison, show signs of reluctance to give up the arbitrary power over the freedom of others that they had been handed in 1940.

After the war, it is very hard to see the justification for the expensive, intrusive and oppressive surveillance that was undertaken. What was true of fascists in the 1940s and 50s is true of suspected Islamic extremists in 2016. There may be moments, like May 1940, when detention without trial in a democracy can be justified, but they are rare and brief. There may be circumstances when spies can be let loose among us, but if it becomes normal, they do far more harm than good. That may well have been the case with my father. The question of who watches the spies should have been important then. It certainly is now.

• Francis Beckett’s Fascist in the Family will be published in September by Routledge