Pastor Fred Phelps is gone, called to glory if you believe the teachings of his hate-spewing ministry, the Westboro Baptist Church. To me it seems more likely that his remains are mouldering away somewhere, obeying the laws of physics and biology. But, either way, it seems an appropriate moment to reflect on the man and his legacy.

I had some history with "Gramps", as his family and followers liked to call him. I made two documentaries about his church for the BBC: The Most Hated Family In America in 2006 and America's Most Hated Family in Crisis in 2010. In all, I suppose I spent about a month with the members of the WBC, trying to figure out what induces them to dedicate their every spare moment – when they aren't holding down respectable jobs as lawyers, correctional officers or salespeople in their hometown of Topeka, Kansas – to flying around the country, standing as close to funeral-goers as they are legally allowed and waving hate-filled placards with slogans such as "Thank God for Dead Soldiers", "Fags Eat Poop", and, of course, "God Hates Fags". They became notorious for picketing the funerals of soldiers killed in Iraq and Afghanistan. In the WBC teachings, the soldiers were being punished for fighting for a nation doomed in the eyes of God for its tolerance of homosexuality.

Their main scriptural inspiration is the passage in Leviticus that mandates the death penalty for gay sex ("Thou shalt not lie with mankind as with womankind: it is an abomination") though, for some reason, the adjacent verses, which proscribe astrology in similar terms, never seem to excite the WBC quite so much. Not to mention that Christ had nothing to say on the subject of gay sex or shouting at funerals and plenty to say about kindness and humility.

The WBC has tended to be a family affair, overwhelmingly made up of Gramps' lineal descendants and their spouses. They live in suburban Topeka, in a collection of houses with connected gardens, which they call Zion. Gramps was the prime mover behind the practices of the church. He founded it when the idea of abominating sodomites was mainstream in American Christian circles. In some respects, it was the times that changed, leaving the WBC behind in their dogged adherence to old-style fire-and-brimstone Bible-thumping. But it's also the case that homosexuality seems to have been an obsession with Pastor Phelps.

According to legend, the WBC inaugurated their anti-gay pickets when a Topeka park became a cruising ground in the 1980s. The Phelps decided to make signs and demonstrate against the practice. The WBC doctrine evolved into a belief that the whole of America was fallen and damned in God's eyes, as was anyone who fought under the US flag – or, indeed, who wasn't a member of the Westboro Baptist Church. We are all either "fags" or "fag enablers" – you, me, Desmond Tutu, Princess Di, Donald Rumsfeld, Billy Graham, Liz Taylor – though possibly not Robert Mugabe: Gramps had a soft spot for him. An eternity in hell is the fate of anyone who doesn't get baptised into the WBC and travel the country waving hate-filled placards at political events, colleges and places associated – even in the most tortuously oblique way – with tolerance of homosexuality.

While I was with them, they had a regular local picket of a hardware store that sold Swedish vacuum cleaners. The Swedish government had imprisoned a pastor for homophobic preaching, and for the WBC that made the store a legitimate target for a ritualised Biblical smackdown. For the newcomer, these pickets were bizarre, not simply because of the venom of the signs, but also because they clashed with the banality of the family interaction. For the Phelpses, it was another day at the office – there was a water-cooler ambience of chit-chat. Meanwhile, everyone, even the youngest child, was carrying placards saying: "Thank God for 9/11", "Your Pastor is a Whore" and "Fag Sweden".

There is no question that their caravan of religious bigotry has made life miserable for thousands of people, many of them vulnerable mourners hoping to pay tribute to recently departed loved ones. Among their proposed picketing targets was the funeral of young Amish children who had been shot by a deranged gunman. In the tortured logic of the WBC, those kids died because their parents weren't out holding pickets denouncing homosexuality. In the end, the WBC called off the event only after they were promised airtime on a local radio station, effectively holding the community to ransom.

But the WBC also made life miserable for themselves and inflicted a distorted and poisonous view of the world on the youngest members of their own family, holding over their heads the threat that any deviation or failure of commitment (not going to a picket or socialising with outsiders) would result in a lifetime of banishment. Ex-members – of whom there are quite a few – can have no contact with the church.

Louis Theroux with members of the Phelps family during the making of his first documentary. Photograph: BBC

Given their eagerness to court controversy, it's not surprising that there are misapprehensions about the WBC. Unlike hate groups such as the Ku Klux Klan, the WBC members never claim to hate gay people themselves, only that God does. I'm pretty sure there was at least one gay man in the congregation of the WBC. Even on the pickets, the Phelps family members could be civil. For most of the Phelpses, the hostility they expressed was a role that they enacted, dictated by a doctrine they had imbibed from their church leader and paterfamilias. You can find videos on YouTube of counter-demonstrators having cordial chats with Phelps picketers. I don't doubt that, if you knocked on the door of one of the second generation of the family, said you had some questions about Jesus, they'd let you in and maybe offer you a glass of water. Pastor Phelps was a different story: he was a hater by instinct.

I'm proud to say he took against me from the moment we met. I asked him how many children he had. He disliked this question – I think he found me trivial. The interview was cut short. Over subsequent days, we continued filming but I hardly saw him. I had the feeling he was hiding from me. We eventually crossed paths again, in church one Sunday after his sermon on the subject of America's coming tribulations, in which he bellowed: "You're going to eat your babies!" One-to-one, Gramps still had the remnants of a folksy, plainspoken charm, but underneath was a bitter contempt for humanity in general and me specifically. I asked him how he could possibly know that the WBC members were the only people bound for heaven. "I can't talk to you – you're just too dumb," he said. It seemed that I was a hellbound sinner. Well, at least I was in good company.

I've heard people speculate that Phelps had repressed gay leanings or that perhaps he was sexually assaulted when he was young, leading to a lasting animosity to homosexuality. Personally, I doubt it. I suspect he had a lasting dislike of the military, which partly explains the picketing of funerals. But there may be no simple explanation for his behaviour. He was just an angry, bigoted man who thrived on conflict. There are credible reports from his disaffected offspring (four of his 13 children left the church) that he was physically abusive to his wife, Marge; he was violent to his children and had an intermittent problem with pills. He was also a lawyer and won some civil-rights cases, receiving an award from the NAACP. But he liked going against the grain.

Members of the Phelps family protesting outside Arlington National Cemetery. Photograph: MCT via Getty Images

The members of the WBC like being attacked for their activities. They thrive on the presence of counter-demonstrators – the patriotic bikers who would sometimes turn up and rev their engines to drown out the WBC's songs at military funerals and also the students who turned out in droves to sing and register their dissent when the WBC held pickets near their campus. For the church, this meant they were getting a reaction and they would quote Bible verses to the effect that being hated by the world was a sign of godliness. Indifference was harder for them to deal with, although they have faced plenty of that as well without being much deterred.

It has been reported that Pastor Phelps had been "excommunicated" from his own church before he died (probably this doesn't mean much more than being prevented from preaching; I doubt he was out wandering the streets). In 2010 I heard a similar rumour. Then, the word was that Gramps was panicking about a multimillion-dollar lawsuit brought against the church by the family of a dead soldier whose funeral they had picketed. (The WBC won the case on appeal.) The rest of the church viewed Gramps' failure of nerve as evidence of lack of faith in God's plan and they put him on the naughty pew for a time-out.

The truth is, despite being its founder and main preacher, Gramps had been a marginal figure within the WBC for some years. When I made my documentaries, the dominant force was Fred's daughter, Shirley Phelps-Roper, a gifted organiser who could sling religious obloquy while holding four separate placards and wearing a bandana with a message of religious hate – in a different context, it would have been impressive. In fact, underneath her programming, and despite all the pain she inflicted in the name of her religion, she is basically a kind person.

Shirley Phelps-Roper on a demonstration with her son, Luke. Photograph: Sipa Press

But my sense is that Shirley has been pushed aside by an axis of WBC men, among them her brothers, Tim and Jonathan, and also the WBC convert Steve Drain, with Steve possibly in the driving seat. This is speculation on my part, but it struck me when I spent time among the WBC members that Steve was the most likely to take over the church. Steve had originally come to the WBC to make a documentary (called Hatemongers) and ended up moving in and bringing his wife and two daughters from Florida. It was striking that he too called Pastor Phelps "Gramps". He had become disconnected from his own parents and found a surrogate family in the Phelps clan. Steve is an intelligent man but arrogant. In personality, he is closer to Pastor Phelps than any of Gramps' natural children. I met and interviewed all three of Pastor Phelps' sons who remain in the church: they all have the slight air of being survivors of an abusive upbringing.

Where the WBC goes from here is anybody's guess. I haven't been following its doings as closely in recent years. Evidently they have attracted new members from outside the family. A few years ago there was news that a US marine and his family had been baptised into the church. Just as striking was the report that a British man had moved to Topeka from England, joined the church and married Jael Phelps. A few weeks ago I found a photo on Twitter of Jael at a picket holding a tiny baby. In its abundant procreation, the family has a guaranteed supply of future recruits.

I don't expect huge changes with Gramps' death. The church has always operated according to the dynamics of a large family rather than a cult. Cults don't typically excommunicate their charismatic leaders. Families do: they put their ageing parents in a granny annex and take away the keys to the car. Maybe, as with other families, the bereavement will bring them together. In another context, that might be a comforting thought. In this case one rather wishes that the second generation would continue to feud and fragment – and perhaps in the process moderate their way of thinking and get in touch with some of the apostate children they no longer see.

The more chilling thought is a backward-looking one, of how one man's legacy is likely to continue. Gramps' offspring, and their offspring, have been raised to believe that abuse is kindness. The natural bonds of family have been braided into this twisted thinking so that children who love their parents and siblings can't separate those feelings from their sense of obligation to the church and its creed. And when they leave they also take with them the nagging guilt and fear that haven't just lost a family: they have lost their only chance of salvation.

• Louis Theroux's LA Stories, Sundays, 9pm, BBC2

• This article was amended on 26 March 2014. An earlier version said Phelps had attended West Point military academy. Although he was admitted to West Point he decided instead to enrol for ministerial training.