Whenever Steve Fulcher is asked to explain his actions during four excruciatingly tense hours on Thursday 24 March 2011 – which he often is by former police colleagues, official enquiries and now journalists – he always turns the question round. “If you were in my position, what would you do?” he asks. It’s a decent point. Same situation, what would you do?

Sian O’Callaghan, a popular 22-year-old from Swindon, had been reported missing the previous Saturday, five days earlier. She went out for the night and never came home. Fulcher, then a senior investigating officer for Wiltshire police, was put in charge. It would soon involve five force areas and 1,000 police officers, not to mention many hundreds of friends and locals who scoured the Savernake Forest, close to the last mast her mobile phone had pinged.

Ultimately, suspicion fell on a 47-year-old taxi driver called Christopher Halliwell, who had picked up Sian during the early hours of Saturday morning, outside a pub. But there was no body and only circumstantial evidence to connect him to her disappearance. After all, couldn’t he have just picked up and dropped her off as he did with paying fares dozens of times every day?

Fulcher didn’t think so and when Halliwell was seen under surveillance buying an overdose-quantity of pills on the morning of 24 March, he ordered his arrest. Halliwell was formally charged in an Asda car park with the kidnap of Sian, and declined to make any comment. So far, so police procedural. But here’s where Fulcher tossed away the script. Instead of bringing Halliwell back to Gablecross police station in Swindon, Fulcher told his officers to take him to Barbury Castle, an Iron Age hill fort on the Ridgeway route.

There were two reasons for this unconventional order. If Sian was alive, Fulcher believed that immediate further questioning was the best – indeed only – way to protect her life. This, he felt, was the directive of Article 2 of the Human Rights Act: the victim’s life came first. And there was a second, more nebulous justification. Fulcher had never met Halliwell and he wanted to look him in the eye. After 28 years in the force, he still believed there was a place in investigations for the policeman’s “gut”.

Life lost: Sian O’Callaghan, 22, was murdered by Christopher Halliwell in 2011. Photograph: Wiltshire Police/PA

What happened next sounds like it was lifted from a TV drama. “It was probably the most intense four hours I’ve ever experienced,” recalls Fulcher, “or I am ever likely to experience.” Under clear blue skies, Halliwell and Fulcher smoked cigarettes, and slowly some kind of connection formed between the detective and the accused. Repeatedly, Halliwell refused to be drawn, but Fulcher chipped away, telling him that if he didn’t co-operate, he would be vilified. The press could be relentless, he warned; just look at the innocent Christopher Jefferies, who had become a front-page punchbag after the Joanna Yeates murder a couple of months before.

Fulcher admits he was only seconds from giving up, but suddenly Halliwell cracked and offered to take him to where he’d dumped Sian’s body. This led them to the prehistoric White Horse in Uffington, further up the Ridgeway in Oxfordshire. Here, as the pair sat in the sunshine, almost on the tail of the chalk horse, Halliwell exhaled smoke and said flatly: “Do you want another one?”

In many ways, it was virtuoso policing. Fulcher had elicited a confession from a cold-blooded murderer who had evaded police scrutiny for decades – a serial killer who studied forensics so he would be better able to cover his tracks. It was work that would see Fulcher nominated for the Queen’s police medal.

But, even as Halliwell was leading Fulcher to where he had dumped the body of Sian O’Callaghan and then to a field where in 2003 he had buried Becky Godden, a 20-year-old sex worker, the policeman knew there was trouble ahead. “Our bond had the flavour now of mutual assured destruction,” writes Fulcher in his new book, Catching a Serial Killer. “We’d both put ourselves into a position where we were sacrificing our lives.”

What Fulcher already knew was that his informal chat with Halliwell was breaching Code C of the Police and Criminal Evidence Act (Pace), which was instituted in the early 1980s in the aftermath of the Brixton riots. Code C specifically protects the rights of an individual against the police with regard to questioning and detention.

Becky Godden, 20, was killed by Christopher Halliwell in 2003 but her body lay undiscovered for eight years. Photograph: Wiltshire Police/PA

Although Halliwell had already been cautioned twice, Fulcher should have done it again. He should also have reiterated the fact that Halliwell had the right to speak to a solicitor. So why didn’t he? “Because I’m holding him on a knife edge,” says Fulcher now. “He never said: ‘Right I’m going to tell you exactly where she is and what I’ve done, etc.’ That isn’t how it worked. All he said was: ‘Have you got a car? We’ll go.’

“It’s a simple moral issue,” he continues. “I did these things because they were the right things to do in these circumstances. In fact, they were the only things to do.”

It would prove to be a costly decision for Fulcher. In 2014, the Independent Police Complaints Commission described his actions as “catastrophic” and he was found guilty of gross misconduct. In court, Halliwell’s barrister, Richard Latham QC, argued that Fulcher had “gone back to the 70s” style of policing. Halliwell claimed that the police officer had threatened his family, so his confession was elicited under duress. The evidence Fulcher obtained about Becky Godden’s disappearance was ruled inadmissible in court.

Although not technically fired, the stress led Fulcher to take leave from work and he began taking antidepressants and sleeping pills. He received strong backing from the families of Sian O’Callaghan and Becky Godden – the latter’s mother, Karen Edwards, launched a petition to have Pace changed to offer greater protection for police officers, such as Fulcher. But in 2014, Fulcher resigned from the service. He has since been, in his words, “unemployable in the UK”.

So it’s a fair question: what would you do if you were in Fulcher’s shoes that day? “But for my intervention, Christopher Halliwell would be walking the streets now,” says Fulcher. “And whatever the public thinks about it – and I’m as liberal as the next person – there’s a very good chance he’d have killed other women in the meantime.”

Hunting the killer: police search the area where Sian O’Callaghan’s body was found. Photograph: Rex Shutterstock

Fulcher was a lifer in the police force. He joined aged 21 and ascended smoothly through the ranks: constable to sergeant to inspector and then, in 2007, to detective superintendent, the highest rank for a boots-on-the-ground police officer. When we meet for a hot dog and beer in London, I ask Fulcher if he had ever been professionally reprimanded before the Halliwell episode. “No,” he replies, “all I’ve ever been is commended. I’ve more commendations than you can shake a large stick at.”

This bears out: Fulcher had received three crown court recommendations and one chief constable’s commendation; there had been no previous findings of misconduct. When a trio of chief constables met to decide his fate, they acknowledged: “His record to date has been unblemished.”

Still, there is something raffish about Fulcher, who is now 50, that doesn’t entirely fit with the picture of the by-the-book copper. He used to drive to work in a gunmetal grey Alfa Romeo Spider with a flashy red leather interior. Some police officers would head to the gym when their shift finished; Fulcher preferred to watch rugby with a couple of pints. “I’ve been described as a maverick,” he sighs, “which I reject, actually.”

How so? “There are two tracks in policing,” says Fulcher. “There’s the track that takes you through to chief constable level and to do that you need to avoid being a detective for any length of time, certainly avoid cases like this. Theresa May, who was a disaster as a home secretary, absolutely loathed the police, but particularly chief constables. There’s a good reason for that, because they are self-serving and hopeless – generally speaking.”

Hell, it appears, hath no fury like a police officer scorned. Much of this ire has to do with the limbo he was left in while the IPCC decided his fate. “It had a devastating effect on my family, as I was in a dark place for a long time,” he admits. “It’s been a horrendous experience, and I really feel for people like Paul Gambaccini [the BBC radio presenter wrongly arrested during Operation Yewtree, the investigation into historic sex offences]. And actually I think that I’d be a far better police officer now understanding the pain, because when we arrest people we might have them on bail for a period of time and they go through sheer hell.”

Mug shot: Christopher Halliwell had evaded police scrutiny for decades. Photograph: Wiltshire Police/PA

Fulcher is currently employed by a private company in Mogadishu, Somalia, as a consultant, subcontracted to deliver aid from Foreign Office and DFID programmes outside the green zone. It’s precarious work, but Fulcher was not exactly inundated with job offers. He says, “I had people phoning me to offer me a job and when they do due diligence, which only consists of Googling me at the end of the day, not unreasonably they say: ‘Look, I’m sorry, but we’re not going to take this on. Why would we?’”

In 2016, a judge, Sir John Griffith Williams, decided that the evidence collected by Fulcher about Becky Godden could and should be heard in court. As a result, Halliwell was found guilty by a jury and his sentence increased to life imprisonment, meaning that he will never be released.

Writing Catching a Serial Killer has been cathartic and also allows Fulcher to set out his thought processes methodically and furthermore to clear his name. With its microscopic analysis of a single crime, the book has the gripping allure of long-form podcasts, such as Serial.

Beyond his personal circumstances, though, Fulcher is furious about what he sees as regrettable and avoidable problems with British policing. Perhaps his biggest complaint is the failure to investigate further crimes that Halliwell may have committed. Police discovered in excess of 60 items of women’s clothing buried in woodland near Marlborough, including Sian’s brown New Look boots and Becky’s cardigan.

“If I’m right, he had a propensity for killing on average once a year,” says Fulcher. “If I’m right, well I take comfort from the fact that he hasn’t been able to do that since 2011. So he definitely killed Sian in March 2011, he definitely killed Becky early 2003. You’re seriously telling me there’s nothing in between or either side? It doesn’t even sound true. Why would this not be the biggest, most protracted, most vigorous investigation in the history of British policing?” he exclaims. “It isn’t and it hasn’t been. Can you have a police force that simply ignores a serial killer?”

Night out: Sian O’Callaghan, 22, with her boyfriend Kevin Reape. Photograph: Rex Shutterstock

Fulcher would also like to see amendments made to Pace that take into account exceptional positions that investigators sometimes find themselves in. “What I do accept is that you cannot have a situation in which every police officer feels that he can breach Pace and point to me and say: ‘Well, Steve Fulcher did it.’ But these are very explicit circumstances, recorded in detail at the time and presented fully, transparently and accurately by me to a court, laying myself open to what did happen to me, which is dismissal.”

Although he is adamant he is not bitter, Fulcher does struggle to keep his emotions in check. And he has a warning. “At some point somebody is going to have to explain to the public what they can expect from the police service when their daughter goes missing,” he says. “And in general, right now, we will not get your daughter back.”

Fulcher wearily shakes his head, “We won’t get your daughter back.”

Catching a Serial Killer by Stephen Fulcher is published by Ebury at £7.99. To order a copy for £6.79, go tobookshop.theguardian.com