On Saturday 23 June 2012, David Miller received an angry email. Miller, a professor of sociology at Bath University, runs something called the Powerbase website, which records the political and business connections of influential people. The writer of the email, who seemed to be based in Russia, explained that “all the content” about Nigel Oakes, a PR fixer and political consultant, published on the website, was “fake” and demanded that it be taken down.

Miller asked for the inaccuracies to be pointed out so that he could make corrections, only for Oakes himself to make contact, a rare event given that, according to ex-colleagues, the old Etonian likes to cultivate an air of mystery. “Nice enough, bit machiavellian,” one said.

An ex-boyfriend of Lady Helen Windsor, Oakes once ran a mobile disco before joining the Tories’ favourite advertising agency, Saatchi & Saatchi.

Powerbase had suggested that Oakes’s claim to “have studied psychology at University College London”, made on one of his firm’s websites, was suspect. Miller has a letter from the university saying it has no record of him studying there. But in his email to Miller, Oakes admitted that while everything on Powerbase about him was true, he wanted it made clear that he had established something called the Behavioural Dynamics Working Group.

The group worked with two respected psychology professors, Adrian Furnham and Barrie Gunter, on understanding and potentially changing people’s behaviour. And it is the key to understanding how a former DJ ended up launching a network of behavioural-profiling firms whose expertise has been eagerly sought by politicians, defence ministries, Brexit campaigners, and now even the US president.

Oakes told Miller that he had been keen to employ the group’s methodology “and required academics to fill in the scientific blanks to my (largely commercial) knowledge”.

But Furnham and Gunter told the website ItalyEurope24 that they cut their ties with Oakes because he was using them to further his career and making claims that could not be substantiated. A spokesman for Oakes insisted he had always had a good working relationship with both men.

The setback did nothing to diminish Oakes’s taste for psychological profiling, and in 1992 he set up a firm that specialised in influencing consumer behaviour through the use of aromas. “Smells can influence attitudes and therefore behaviour,” Oakes said when he launched Marketing Aromatics. The company appears to have had limited success but, by the late 1990s, Oakes had become interested in more than selling products: he was selling politicians. He established Strategic Communication Laboratories (SCL) Group, which in 1999 won a deal to enhance the reputation of Indonesian president Abdurrahman Wahid.

SCL specialised in influencing behaviour by identifying key audiences and developing strategies to connect with them, and it offered a taste of things to come. Those who visited its Jakarta office told the Independent that it resembled “a Tom Clancy novel”, a vast room with huge screens and a huge one-way mirror behind which men and women sat glued to the internet.

The Jakarta ops centre appears very similar to one built by a company called Vision 360 for the James Bond film Goldeneye. This may be more than coincidence: Vision 360 built a similar control room for SCL when, in a bid to win military contracts, it began exhibiting at the annual Defence and Security Equipment International show.

According to a press release, SCL’s Ops centre “contained a 12-screen media feed, a 4x3m LED presenter video wall, 62 computer screens, a large 6x4m rear-projected screen, plus an enormous 10x4m operations overview screen.” It claimed Oakes was delighted with it.

Afterwards, SCL started winning government contracts, including one from the Foreign Office to help counter violent jihadi propaganda in Pakistan.

As more contracts rolled in, SCL attracted funding from powerful investors including Lord Marland, a trade envoy under David Cameron. Oakes’s ambition for his company grew. In 2007, SCL paid $20,000 to a Washington lobbying company, Global Policy Partners, to promote it in the US. SCL won contracts with the Pentagon to conduct surveys in Iran and Yemen. “Psych-ops was big business and people were just chucking money around,” said one person familiar with the company.

In 2015, it secured a $750,000 contract to help Nato states counter Russian propaganda in eastern Europe.

At the same time it was wooing the military, SCL was using its skills to help win elections. Several Caribbean politicians paid handsomely for its services and its reputation flourished. It spawned another company, Cambridge Analytica, which sought to use “data modelling and psychographic profiling [classifying people into personality types] to … connect with people in ways that move them to action”. Cambridge Analytica forged links with the US right, winning contracts with several Republican governors and working for Ted Cruz on his failed campaign to become the Republican presidential candidate.

According to returns filed with the US tax authorities, in 2014 it received more than $1m from America Inc, whose mission is “to educate Americans about traditional American values”. America Inc’s president is L Brent Bozell III, a Tea Party sympathiser who founded something called the Media Research Center “to neutralise leftwing bias in the news media and popular culture”.

The US billionaire Robert Mercer, a major financial supporter of Donald Trump, has reportedly become a Cambridge Analytica shareholder. Photograph: ddp USA/REX Shutterstock

The US billionaire Robert Mercer – a major Trump supporter who is close friends with Bozell and the former Ukip leader Nigel Farage – was so impressed with Cambridge Analytica that he has reportedly become a major shareholder. Trump’s chief strategist, Steve Bannon, used to sit on its board. Perhaps inevitably, given these connections, the Trump campaign ended up paying almost £5m to the company to help it target swing voters. Mercer introduced Farage to Cambridge Analytica, according to Andy Wigmore, communications director of Leave.EU, the Brexit campaign championed by Farage.

On its website it boasts: “We collect up to 5,000 data points on over 220 million Americans, and use more than 100 data variables to model target audience groups and predict the behaviour of like-minded people.” Its chief executive, Alexander Nix, is an evangelist for the company’s ability to reach voters that ordinary companies can’t.

In an article for Campaign magazine last February, he described how his company had “helped supercharge Leave.EU’s social media campaign by ensuring the right messages are getting to the right voters online.”

But some are dubious about the behavioural prediction claims. Frederike Kaltheuner of Privacy International, the organisation that campaigns against intrusive surveillance, said that she had tested the technology using her own Facebook data. She said: “Cambridge Analytica uses data and machine-learning to profile people and predict personalities and attributes. When I did their test I had a 65% probability of being male, and [was] likely [to be] homosexual.”

How this new generation of profiling companies is collecting data is becoming an issue for privacy campaigners and watchdogs. Responding to the disquiet, the Information Commissioner’s Office has announced that it is to conduct an analysis of the use of big-data analytics, including for political purposes.

A spokesman for Cambridge Analytica said fears about its use of psychographics were misplaced. “There’s nothing magical or Pied Piper-ish about it. It doesn’t give us special powers over people. We’re all trying to better use the behavioural sciences to do our work in more effective ways.”

Indeed, Cambridge Analytica is just one of many new companies trying to reach voters in novel ways. Both Barack Obama and Hillary Clinton’s campaigns employed behavioural profiling companies. The Vote Leave campaign paid £3.5m to a tiny Canadian company called AggregateIQ, which specialises in targeted Facebook advertising and profiling.

Cambridge Analytica’s chief executive, Alexander Nix, at the company’s office on 5th Avenue in New York. Photograph: The Washington Post/Getty Images

“The Vote Leave campaign owes a great deal of its success to the work of AggregateIQ,” Vote Leave’s director Dominic Cummings has said. “We couldn’t have done it without them.”

Puzzlingly, the phone number for AIQ’s president, Zack Massingham, is the same as that of SCL Canada, according to a cached version of SCL’s website, now taken down. SCL explained that it had partnered with AIQ in the past and that the Canadian firm had once acted as a point of contact. “It was recently brought to our attention that their contact details were still on the company website, so we updated the page by removing them,” SCL said.

Massingham, who confirmed SCL’s account, told the Observer his company’s focus was about engaging with a campaign’s supporters and tailoring the message to them. “It’s about communicating with them in a timely and meaningful manner, not giving them too much content, and what you do give them is within the lines of what they want to hear.”

A debate is raging. Can behavioural profiling influence elections or is it a hyped technology, albeit with major consequences for privacy? Either way, the industry has come a long way since the 1980s when it first fascinated Oakes. “Behavioural modelling involving big-data analytics has arguably passed an inflection point,” said Dr Simon Moores, an expert on cybersecurity. “Thanks to the growth of predictive analytics, algorithms and big data-mining businesses you can now look forward to a future that’s made up of equal parts Orwell, Kafka, and Huxley.”