A few years ago, Paul Hawkes was imprisoned in a Bavarian castle. He escaped by fashioning a cord out of curtains and sheets, which he used to abseil out of the window. Friends joked that this was just another day in the life of a private investigator, but Paul’s captors weren’t criminal masterminds – they were hospitality staff. Having hosted a party for Paul and his friends on holiday in southern Germany, they had accidentally locked him in and gone home.

When I visit Paul at the offices of Research Associates, the “investigation and intelligence agency” he runs in Notting Hill, west London, he is seated behind a desk with a magnifying glass in front of him. PIs occupy a special place in the British psyche, conjuring up a range of images that runs the gamut from the svelte genius of Sherlock Holmes to rogue investigators hacking the phones of dead children. I ask Paul to recall his most theatrical case but he struggles to think of one. He remembers having his cover blown and falling foul of a Russian arms dealer he was investigating for fraud “somewhere around 1992 or 1993”. “I certainly don’t like arms dealers being angry with me,” he says.

Paul tries to maintain a sense of humour about his work and has nicknamed his office manager “Moneypenny”. Victoria, as she prefers to be known, asks me for a secret password when I arrive. It’s a joke – apparently – but one that highlights the industry’s complicated relationship with its own mythology.

“Every piece of kit I’ve got, you can now get on the internet,” says Richard Martinez, founder of the Expedite Detective Agency on the outskirts of Croydon. “Ten years ago, you couldn’t get hold of a spy pen [a pen fitted with a microphone and camera] and now housewives can simply go on Facebook to get all the information they need.”

The office doubles as Richard’s home, and business has been slow of late. “My competitors may say they’re a lot busier,” he says. Most of Richard’s trade still comes from marital investigations. He once trailed a game-show host with a string of houses dotted around the M25. One was for his wife and children. Two housed mistresses. The man used the fourth to rest in after visiting the other three. Richard says nine times out of ten his clients were correct to worry.

He lays out a selection of gadgets and equipment on his dining room table: the tools of the trade. They include a lie detector, a pair of glasses fitted with a video camera, a spy pen and a fake beard. “There was a make-up artist who had an excellent line in facial hair from goats,” he says. She had “a skin glue” to stick the animal hair to his agents’ faces.

Richard offers clients a “honeytrapping” service as a sort of pre-emptive strike to help them avoid future marital problems. For a fee, Expedite can have someone flirt with your partner and offer them their phone number, usually in a bar, to discover the likelihood of them cheating.

“About 50 per cent would actually fail,” Richard says.

He mentions having to brush off accusations of entrapment, a subject he covers in “Honeytrapping (the Sexpionage Song)” – his track about life as a PI, available to buy on iTunes – in which he extols the superior logic of paying someone to chat up a partner who might be cheating, subtly implying that they almost certainly are. I left with a free copy of his autobiography, Honeytraps and Sexpionage, and with the sense that he was doing quite a bit to emphasise the sensational side of private investigation.

Paul Champion, the president of the Association of British Investigators, is doing the opposite. He says that marital investigations of the sort associated with PI cases are becoming a thing of the past. Fieldwork has given way to office-based research – and yet still “people conceive a Philip Marlowe-type character who stands in a doorway in the pouring rain with a camera and dirty trench coat”.

Most PIs I approached refused to be interviewed, fearing that I would perpetuate this kind of false image. Others were less concerned. Above a shopfront in Crystal Palace is a sign bearing the letters “FBI”. That’s Finlay’s Bureau of Investigation, not the south London outpost of the US federal bureau.

“I became a PI to help my mum,” explains its director, Steve Finlay. “Personally I’m into maths and physics. I’m not attracted to the business at all, but my mum’s pension is tied up in the company. If I stop, she’ll lose it.”

I ask what inspired the name. “We have no aspirations of taking on the FBI,” Steve says. “They are a totally different business. The name is just a name and my dad realised people would remember it, especially 50 years ago.

“Sorry if this all sounds boring,” he says. “You may do better writing an article on solicitors’ clerks – because that’s basically all we are.”