A little before lunchtime on 19 September 2011, a debonair academic was preparing to give a talk in London at a thinktank overlooking the Thames. Dr Robert Lambert, a tall, well-dressed man in his late 50s, delivered an hour-long lecture about his newly published book, charting selected parts of his 26-year career in special branch. The book made no mention of the darker periods of his past. But within a few months Lambert's reputation would be in tatters.

Lambert joined the police in 1977, aged 25. Within three years, he was in special branch, and soon after he was recruited into the Special Demonstration Squad, a top-secret unit within London's Metropolitan police.

His undercover persona was Mark "Bob" Robinson, a charming, intelligent radical with a taste for danger. In 1983 – the first year of his deployment – Lambert met Charlotte, 22, at an animal rights demonstration outside Hackney town hall in east London.

"He told me that he worked as a gardener in north London," Charlotte says. "Wherever I turned, he was there trying to make himself useful, trying to get my attention." Lambert was Charlotte's first serious boyfriend, and he gave the impression of being a dedicated political activist. "He would tease me for not being committed enough," she says. "I was a vegetarian, but he encouraged me to become a vegan and he got me to become more involved in direct action."

Within a few months, the pair were an established couple among radical protesters in London. "Although Bob had a bedsit, he would stay with me. He would sometimes go off for a short while, saying he had to visit his dad with dementia in Cumbria, and sometimes he had a gardening job. Most of the time while we were together, he lived with me."

It was a double life. Lambert's father did not have dementia and did not live in Cumbria. His periods away from Charlotte were spent with his wife and children in Herefordshire. For at least five days a week, however, Lambert was with Charlotte.

One of the hardest challenges for covert officers is turning up out of the blue without friends or family to vouch for them. Acquiring a girlfriend is an easy way to fill the gap, making an undercover police officer seem like a real person. "One day, Bob wasn't there," recalls a friend of the couple. "And then he was everywhere."

Soon, Lambert was throwing himself into political activity, becoming involved in squatting, free festivals and anti-nuclear weapon camps. He became interested in a small, radical environmental group called London Greenpeace, which bore no relation to the larger campaign with the same name. It was the next step towards what had become the main objective of his mission – penetrating the intensely secretive hardcore wing of the animal rights movement, the Animal Liberation Front.

Lambert set about befriending campaigners suspected of being in the ALF. One was called Geoff Sheppard. Just as he had done with Charlotte, Lambert made Sheppard feel the pair had a special connection: they were locked together in the struggle. "I believed in him, and I liked him. I thought he was a friend of mine," Sheppard says. According to him, Lambert even produced a well-known ALF leaflet from the era that summed up the group's philosophy.

Sheppard recalls the moment in 1987 when a trio of ALF activists concocted a plan to set fire to three branches of Debenhams in an attempt to force the department store to abandon its fur products. His testimony about the attack on Debenhams – and the part he alleges Lambert played – was highlighted in a parliamentary speech by Green MP Caroline Lucas in June 2012. Lucas told the house that Sheppard had said in his testimony: "There's absolutely no doubt in my mind whatsoever that Bob Lambert placed the incendiary device at the Debenhams store in Harrow."

Lambert has consistently denied planting the device in the Harrow store, which cost the company an estimated £340,000, but takes credit for jailing Sheppard and Andrew Clarke, the other ALF activist who would ultimately be convicted of the attack. For another quarter of a century, despite years in jail, Sheppard never once considered that his friend Bob Robinson had betrayed him.

Two years before the arson attacks, in the autumn of 1985, Bob Lambert had been standing in a hospital holding his newborn son. In a nearby bed, Charlotte lay recovering. "Bob was there by my side through the 14 hours of labour," Charlotte says. "He seemed to be besotted with the baby. He was a great dad and I had no reason to believe that our son was not his first. I didn't realise then that he was already married with two other children."

Helen Steel lived with undercover officer John Dines for two years. ‘I fell madly in love with him, in a way I never have since. He said he wanted to spend the rest of his life with me.’ Photograph: Ed Thompson for the Guardian

Lambert was not the first SDS officer to father a child in the field. At least one other child had already been born to a member of the squad in the early 1980s. Rather than receive any reprimand, that SDS officer was later promoted to a senior post in the squad. But, on the whole, fathering children was not what police spies were supposed to do. It made life complicated.

Because Lambert and Charlotte were unmarried, they were required to sign the birth register together. But Lambert let Charlotte down on the handful of occasions when they made appointments to visit the registrar's office, so Charlotte was forced to register her son under her own name. In hindsight, Lambert's refusal to sign the document looks odd; but at the time it appeared in keeping with the beliefs of a radical activist who eschewed any connection to the state.

It was not an unwelcome pregnancy, however. Charlotte wanted the baby and she got the impression that Lambert felt the same. Initially, he took the child on father-and-son outings and spent most of his spare time with his new family.

But in 1987, at the height of his infiltration of the ALF, Lambert became more distant. One of the perceived strains on their relationship was lack of money. Friends of Charlotte recall how she was initially happy to take the greater responsibility for earning money, allowing Lambert to dedicate his time to politics. But it became a source of friction. Another was that, 18 months after the birth of his son, Lambert was complaining that Charlotte was neglecting their sex life. Charlotte believes that Lambert deliberately provoked her and started wearing her down. "With the benefit of hindsight, I can now see how he orchestrated the breakdown of our relationship."

Charlotte was one of four sexual relationships Lambert had while undercover. A second was little more than a one-night stand, and a third lasted some months. His fourth was curious, because it was not with an overtly political campaigner, but with a woman whom Lambert believed could lend his undercover identity further credibility.

Karen met Lambert at a party in north London in May 1987, around the time his relationship with Charlotte was falling apart. Karen was a 24-year-old who had come to the capital to find work, and was intrigued by Lambert. "I thought I had found my Mr Right. He was very charming and I thought I could take him to meet my parents," she says. Karen was aware that Lambert had a young son from a previous relationship, and he occasionally brought the boy along when he saw her. But on the whole he came across as a free spirit with a politically rebellious streak.

There was a time during the summer of 1987 when Lambert was spending at least one day of the week with his wife and two children in the suburbs, and the rest with either Karen or Charlotte, with whom he was still sleeping. There was a reason for Lambert to maintain ties with both women. Every SDS officer needed a plausible excuse to drop everything and disappear – and it was important that there were people close enough to them to vouch for their vanishing act.

Following the arrests of Sheppard and Clarke, Lambert told Karen, Charlotte and other friends that he could be next in line to be picked up. Over the last few months of 1988, he and Karen discussed what to do. It appeared obvious that he had to make himself scarce for a few years.

"I was heartbroken," she says. "Even when he left, I could not imagine that it had finished, because we loved each other so much. I wanted to go on the run with him." In early 1989, Karen received a long letter from Lambert postmarked Valencia, Spain, saying he was not coming back but raising the possibility that she could join him there. It was the cruellest of false hopes, but Lambert knew it would make his disappearance seem more genuine.

Earlier, he had been having similar discussions with Charlotte. "He promised he would never abandon his son, and said that as soon as it was safe, I could bring our baby to Spain to see him." Charlotte, too, received a letter from Lambert from Spain. It was the last she, her son or Karen ever heard from "Bob Robinson".

Eventually, Charlotte began a relationship with another man and married him, but after just five years her husband died. Her son, then eight, had now effectively lost two fathers. Distraught, Charlotte became desperate to find Lambert, believing he could help their boy. She enlisted the help of social services and the Child Support Agency, but time and again official records drew a blank. It was as if Bob Robinson didn't exist.

At that time, Lambert was just a few miles away, behind a desk at Scotland Yard. He eventually left the police in 2007, after managing dozens of undercover officers. It was with some astonishment that fellow officers then watched him assume a public profile as an academic. He took on postings at St Andrews and Exeter universities, and became a regular fixture on the speakers' circuit. He even appeared on television.

In the end, it was veteran activists from the now-defunct London Greenpeace who realised, in 2011, that Bob Robinson was not a fugitive still hiding in Spain, but an academic touring lecture theatres in Britain. There followed a series of revelations about Lambert's secret past, including, in June 2012, a Daily Mail piece about the Debenhams arson attacks.

Thursday 14 June had been an ordinary day for Charlotte. "I came home from work at about 4pm. I made a pot of coffee and, because the weather was good, I took the Daily Mail and the coffee out to the garden. As I flicked through the paper, I saw the picture of Bob Robinson in the 80s – it was 'my' Bob, my son's dad. I had not had news of him for approximately 24 years and there was his face staring back at me. I went into shock. I felt as if I couldn't breathe and I started shaking."

Charlotte spent the next day trying to track down Lambert. She knew he was now an academic at St Andrews. "I called the university and was put through to a woman in his office." Ten minutes later, the phone rang. "It was Bob," she says. "This was the first time I had heard his voice for 24 years, but I recognised it. It was very emotional. I remember asking him, 'Why me?' " She says Lambert sounded emotional, but failed to fill the gaps. "He could not answer my questions," she says. "I could no longer believe a word he said."

The trauma of discovering Lambert was a police spy led to months of psychiatric treatment. Friends say Charlotte has not been the same since. She is constantly on edge and has had suicidal thoughts. "I feel so confused and hurt by what has happened," she says. "I don't understand what I am supposed to have done that I was chosen by the state to be treated like this. I was no threat to national security. And what was my child – collateral damage?"

John Dines had taken the identity of a boy who died at eight. ‘The discovery was like a bereavement. Suddenly he didn’t exist. All my memories are of a nameless stranger’

Lambert's deceptions of Charlotte, Karen and his son were not the only skeletons in his closet. His time undercover had coincided with the epic legal battle known as McLibel. The case involved a tiny environmental group that produced a roughly typed leaflet castigating the world's biggest hamburger chain, McDonald's. Instead of ignoring what was little more than a pinprick in its reputation, McDonald's executives decided to exploit England's notorious defamation laws and sue the activists for libel. They presumed the campaigners would bow to their demands, withdraw the leaflet and say sorry. Against all expectations, two stood their ground and took on the corporate power in what turned out to be England's longest ever civil court case.

The group behind the McLibel leaflet was London Greenpeace. And one of those responsible for writing the offending leaflet had been Bob Lambert.

The police officer was not the sole author of the leaflet but, according to several key members of the group at the time, he co-wrote it. "He was really proud of it," one of Lambert's friends recalls. "It was like his baby – he carried it around with him." Paul Gravett, an activist in the group, says that while several people had input into the leaflet, Lambert was "one of the most prominent people in the group at the time". Lambert even confided in his then girlfriend, Karen, that he was behind the leaflet, although he appeared more reluctant to admit as much around others. "He did not want people to know he had co-written it," she says. "He did not want to draw attention to himself."

Lambert was not the only SDS spy to infiltrate London Greenpeace. As his deployment came to an end, senior officers at the SDS decided to send a second operative into the tiny group. The spy they chose was John Dines, who went undercover with the alias John Barker and struck up a relationship with Helen Steel, one of the two campaigners who stood up to McDonald's.

Dines started courting Steel in 1990. "He said he wanted to spend the rest of his life with me," she says. "In a short space of time I fell absolutely madly in love with him in a way I had never fallen in love with anyone before or since. He said he wanted us to have kids. He used to say he had once seen an elderly Greek couple sitting on a veranda gazing into the sunset, and that he pictured us growing old like that." By the summer of 1991, as part of an exit strategy, Dines began exhibiting symptoms of a breakdown. "He kept talking about how he had nobody left apart from me," Steel says. "His parents had both died. He had no brothers and sisters. The only woman he had ever loved before me, a woman called Debbie, had left him. He said he was convinced I was going to do the same."

In March 1992 Dines left for South Africa, saying he could not handle things any more. After that, Steel received two letters with South African postmarks. Then her boyfriend vanished altogether: "I was sick with worry that he might kill himself." Steel contacted the British consulate in South Africa and eventually hired a private investigator, who could find no trace of her partner.

In fact, Dines had returned to a desk job in Scotland Yard. But he left the police in 1994 and was given a pension to compensate for ill health. He later returned to New Zealand, where he had claimed to have spent some of his teenage years.

In her search for clues to his whereabouts, one of the first things Steel did was locate a copy of what she believed was her boyfriend's birth certificate. The document confirmed the details he had given her: he was born in Derby in January 1960. It was another 18 months before Steel decided to inspect the national death records. She was astonished to find the real John Barker had died of leukaemia as a child. "It sent a chill down my spine," she says. "When I got the certificate itself, it was so clear. The same person. The same parents. The same address. But he had died as an eight-year-old boy."

The discovery turned Steel's world upside down. "It was like a bereavement, but it wasn't something I could talk to people about. Now, suddenly, he didn't exist. This was a man I had known for five years, with whom I had lived for two years. How could I trust anybody again? All the photographs I've got, all the memories I've got are of a nameless stranger. What do you do with that?"

Clues led Steel to a public archive in New Zealand and it was there, in 2002, that she found a document that linked Dines with Debbie, the woman he had married more than a decade before he and Steel met. Back in London, she ordered the couple's wedding certificate. She immediately recognised her boyfriend's handwriting. "What hit me like a ton of bricks is that he listed his occupation as a police officer," she says. "When I read that, I felt utterly sick and really violated. It ripped me apart, basically, just reading that."

Steel now knew that Dines was a policeman when he got married in 1977. But there was still a possibility he'd given up his job before becoming an activist. She shared the evidence with friends and family. Some cautioned her against concluding Dines had been a police spy. "I remember my dad and others said, 'You're being paranoid – that would never happen in this country.' "

In 2002, when the SDS feared that Steel was getting close to Dines in New Zealand, they took a remarkable decision. At considerable cost to the British taxpayer, they decided to uproot and relocate their former spy to another country.

Jim Boyling had two children with Laura, whom he met while ­undercover. ‘I don’t think the police consider us at all,’ she says

Helen Steel had not been the only woman searching for an invisible man. Laura, an environmental activist, met Jim Boyling towards the end of his undercover deployment as "Jim Sutton", a former hunt saboteur now active in the protest group Reclaim The Streets. This was in the summer of 1999, when they attended an RTS meeting in London. They moved in together, but the romance was almost overwhelming. "In the beginning I nearly broke it off because it felt too strong," Laura says. There was only one moment when she questioned the background of the man she thought was her soulmate. It was the briefest flicker of doubt. "It was the way he was cleaning his walking boots. I suddenly thought, 'Who the hell is in my kitchen?' and then I came to and suddenly he was Jim again."

Then, in May 2000, "out of the blue, he told me that he had to leave me". After he left, Laura began looking into his background. She was worried he might be at risk in some way. She discovered from official records that he was not adopted, as he had said, and neither was he born on the day he claimed. An email led her to believe he was working in a vineyard in South Africa. In the summer of 2001, she spent three months searching for him there. She returned to London, but with nowhere to live. "I used all my savings trying to find him and I was very thin, down to 6 stone 12lb. I stayed for a while in a backpackers' hostel on Gray's Inn Road and on a stranger's sofa."

She eventually tracked Boyling down to Kingston in Surrey, where he made a confession of sorts. He admitted he had been a police spy and disclosed his real name, but claimed that his experience undercover had changed him. He told her he was very much in love with her and wanted to continue the relationship.

Laura says Boyling repeatedly promised her that he would leave the police and start a new life. Within two weeks of their meeting, she became pregnant with his child. They eventually married and moved out of London, now with two children. Laura says she hoped that marriage would bring him stability and the courage to leave the police, but Boyling became "increasingly controlling, erratic and abusive". In February 2007, she entered a women's refuge, after receiving help from the same organisation for more than eight months.

Laura desperately wanted to contact her old friends in the environmental groups, but she had no way of knowing whom she could trust. If her own boyfriend turned out to be a police officer, how could she know who was a real protester and who was an informer?

"You don't expect the one person you trust most in the world not to exist," she says. "I don't think the Metropolitan police consider us at all… You are a head to be trodden on on the way up the ladder of credibility."

For his part, Boyling insists he "never behaved abusively" towards Laura. He says they were "no longer together as a couple" when she went to the refuge. He adds: "Notwithstanding our separation, I have always tried to support her. I have always supported the children financially and continue to do so. Despite everything, I don't wish to be critical of Laura, who has always been a loving mother to our children. Life has been difficult enough."

Two years after her divorce, when still seeking help for the trauma of the relationship, Laura plucked up the courage to contact one of her activist friends: Helen Steel. When they met, Laura told Steel that Boyling had been a spy. She also revealed that he claimed he felt sorry for Steel, as she had been spied on by three undercover officers – himself, Lambert and John Dines.

Steel had been waiting more than 18 years for confirmation. "Although it was massively painful, there was a sense of relief that I finally knew the truth. I didn't have to keep wondering," she says. For nearly two decades, Steel hoped that, despite his betrayal, Dines might have genuinely loved her. It was only recently that she decided his love was also fake. "I got out all the old letters that he sent me and read them again, with the knowledge that he was an undercover police officer and that his parents were still alive," she says. "What had once seemed like heart-wrenching stories in these letters, disclosures that made me really worried about his wellbeing, were completely false. That is manipulation. It is abuse."

Some names have been changed.

• This is an edited extract from Undercover: The True Story Of Britain's Secret Police, by Paul Lewis and Rob Evans, published by Guardian Faber Books at £12.99. To order a copy for £8.49, including free UK mainland p&p, go to guardian.co.uk/bookshop.