From the outside it all looked haphazard and frenzied. A campaign that was skidding from scandal to crisis on its way to total defeat. That’s not how it felt inside the ‘Project Alamo’ offices in San Antonio, Texas where Trump’s digital division — led by Brad Parscale, who’d worked previously with Trump’s estate division setting up websites — was running one of the most sophisticated data-led election campaigns ever. Once Trump’s nomination was secured, the Republican Party heavyweights moved in, and so did seconded staff from Facebook and Google, there to help their well-paying clients best use their platforms to reach voters. Joining them were 13 employees from the UK-based data analysis firm Cambridge Analytica, who were led by chief product officer Matt ‘Oz’ Oczkowski, who had enormous biceps and walked around the office carrying a golf club.

Brad, and his boss Jared Kushner, had bet the house on running a data-led campaign, figuring that was their best chance against the formidable Clinton machine. Cambridge were the data guys brought in to help him do it. Their main job was to build what they called ‘universes’ of voters, grouping people into categories, like American moms worried about childcare who hadn’t voted before.

Cambridge had a database of around 5,000 data points on 200 million Americans and combined it with the Republican Party’s own voter data to build dozens of these highly focused universes and model how ‘persuadable’ its members were. (For example, analysts discovered during the race that a preference for cars made in the US was a solid indication of a potential Trump voter). Creative types then designed specialised ads for these universes, based on the specific things they were thought to care about. Every-thing was tested, retested, redesigned. They sent out thousands of versions of fund-raising emails or Facebook ads, working out what performed best. They tried donate pages with red buttons, green buttons, yellow buttons. They even tested which unflattering picture of Hillary worked best.

It wasn’t just about the online ads, either. Cambridge Analytica’s universes also showed where Trump should hold rallies. A few weeks in, for example, Brad shifted budgets to the swing states of Michigan and Wisconsin. Kushner told Trump to start campaigning in Pennsylvania too. Commentators at the time said that was stupid. But it was all data-led.

Over the last few days, Cambridge Analytica’s use of these techniques has hit a very big democracy-in-crisis nerve. It feels sinister, manipulative, deceptive. A former employee, Christopher Wylie, claimed Cambridge was using millions of Facebook users’ data without the proper permissions (the company denies this). There followed a Channel 4 documentary which caught Cambridge Analytica CEO Alexander Nix bragging about various shady ways in which his company could win elections, including sending in Ukrainian women as honey traps and a litany of other potentially illegal activities. On Tuesday, Cambridge suspended Nix.

Lots of the outrage has revolved around Cambridge’s possible use of ‘psycho-graphics’. This is a specific technique of profiling people’s personality types — openness, conscientiousness, neuroticism and so on — and using those profiles to inform messaging. Cambridge denies using psychographics during the Trump campaign, but has used this technique in the past, and even claimed it could predict the personality type of every single adult in the US.

Important and intriguing as this all is, it’s also a bit of a distraction. It pins the blame on Nix as a silver-tongued Old Etonian villain with megalomaniacal tech gurus. What should be more worrying is that nearly all the digital methods used by Cambridge are both perfectly legal and widespread. Facebook is currently reeling and apologising — it has already lost several billions in market value this week following the Cambridge allegations.

It’s easy to see why the company is worried: its entire business model is based on serving up exactly this kind of insanely targeted insight and micro-advertising. Bear in mind that Facebook still boasts how fabulously and precisely it can target voters: ‘Using Facebook’s targeting tools, the [Conservative] party was able to reach 80.65 per cent of Facebook users in the key marginal seats’ reads one ‘Success Story’ on the company’s ‘Business’ page about the Tory 2015 win. ‘The party’s videos were viewed 3.5 million times, while 86.9 per cent of all ads served had social context — the all-important endorsement by a friend.’

Perhaps we’d be less upset by this if Facebook correctly identified itself as primarily an ad firm, rather than a socially spirited connector of humans. Mark Zuckerberg’s dream of joining the world also means connecting Cambridge Analytica’s data guys to Ford-driving American moms worried about childcare who’d never voted before.

Nearly, if not all, of Cambridge’s data set was legally acquired: buying up consumer records, credit-card data, telephone surveys, and so on. Lots of companies do this. There’s a whole network of data analytics firms using sophisticated cookies and tracking software to follow consumers around the web, or buying commercially available personal data. Thousands of times a day we are put unwillingly and unknowingly into ‘buckets’ or ‘universes’ by clever data analysts who obsess over ‘click-through rates’ and ‘conversion’. Sometimes that’s to sell us a holiday. But it’s useful for flogging us politicians, too.

The modern election is still mostly about having the right candidate. But it’s increasingly important to get the right message to the right people at the right time — tapping into their deepest emotions and fears to figure out what buttons to push. We used to call this sort of thing propaganda. Now we call it ‘a behavioural approach to persuasive communication with quantifiable results’ and give awards to the people who are best at it.

A network of private contractors and data analysts who offer their wares to political parties are spreading these techniques all over the world. Dominic Cummings, who as Vote Leave’s campaign director ran a very data-led campaign, wrote after the Brexit vote that ‘if you want to make big improvements in communication, my advice is — hire physicists, not communications people from normal companies’.

Quietly but effectively, Labour relied on these sorts of techniques in 2017, applying data modelling to figure out potential Labour voters, and then a/b testing them with messages. They used an in-house tool called ‘Promote’, which combined Facebook information with Labour voter data, allowing senior activists to send locally based messages to the right (that is, persuadable) people. Everyone’s at it to a greater or lesser degree. In 2008, analysts working for Barack Obama assigned every voter in the country a pair of scores that predicted how likely they were to cast a ballot, and whether they supported him. In 2012, Google chief Eric Schmidt even advised the re-election campaign, and no one batted an eyelid. Hillary Clinton, too, had an extremely sophisticated system of targeting voters online. I don’t recall liberals being up in arms about this. They seemed perfectly comfortable when it was their candidate on the data rip.

Every election is a mini arms race between warring tribes, which makes it very difficult to slow down the technological advances. Each year the digital tricks get a little more sinister. They dig a little deeper into our skulls and we barely notice. Just imagine what sort of predictive-personality micro-targeting will be possible with the so-called ‘internet of things’. By 2020 there will be around 50 billion devices connected to the net — four times the current figure — and each one will be hoovering up your data: cars, fridges, clothes, road signs, books. Your precious daughter playing with her doll: data point! Your loved one adding some sugar to her tea: data point! Within a decade your fridge will work out what time you eat, your car will know where you’ve been, and your home assistant device will work out your approximate anger levels by your voice tone. This will be gobbled up by hungry political analysts. By cross-referencing fridge data against the number of emotional words in your Facebook posts, Cambridge Analytica or some other strategic communications team will work out that you get angrier when you’re hungry and target you with an emotive message from a law and order candidate just as you’re feeling peckish. Just received a warm message plus donation page link from Caroline Lucas? That’s because your smart bin shows you recycled that morning and a tweet analysis suggests you’re in a good mood.

But the big question is: does any of this stuff even work? Or does it just make the people who pay for it feel as if they can manipulate public opinion? There’s a strong element of Wizardry of Oz about data and targeted advertising. Because so few people understand the science, people credit it with a dark power that isn’t altogether real. People didn’t vote for Trump simply because Cambridge Analytica (or Russian bots) told them to. Far too many otherwise intelligent people, unable to comprehend Trump’s popularity, are convinced that voters were duped. The data companies are happy to propagate this myth because it’s good for business.

Still, in a tight election these sorts of techniques can and do make a difference. Consider this: Trump won Pennsylvania by 44,000 votes out of six million cast, Wisconsin by 22,000, and Michigan by 11,000. These are tiny numbers: less than 1 per cent. If they had gone to Clinton, as widely projected, she would be president. In a close race with two unpopular candidates and a small number of key marginal districts, Cambridge Analytica’s refined universes and Facebook’s targeting wizardry meant Trump could reach enough of the right people in the right districts with the right messages at the right time. So yes, you can argue that Cambridge Analytica did swing it for Trump.

But that’s not what’s most worrying. The shift towards big data elections has profound consequences for the whole of modern politics. If every voter is reduced to a data point who receives not real messages from politicians, but machine--generated adverts finely tuned towards personality and mood, then elections become little more than a software war. And the more politics is a question of smart analysis and nudges rather than argument, the more power shifts away from those with good ideas and toward those with good money or good data skills. (That could be left- or right-wing of course).

Worse still is the fragmentation. Micro-targeting chips away at the idea of a shared public sphere. Instead of open debate each person has their own prejudices and pet projects echoed back at them. Persuasive adverts have always been used in politics. Remember the ‘Labour Isn’t Working’ poster? But big data points to a very different approach: work out who people are, find the one thing they care about, and zero in on that. There are benefits to receiving messages that appeal to you, but it is important that every-one gets the same message — or at least knows what others are getting — because that’s how we thrash out the issues of the day. If everyone receives personalised messages, there is no public commons — just millions of private ones. In addition to narrowing the scope of political debate (research suggests that candidates are more likely to campaign on polarising ‘wedge’ issues when the forum is not public) this diminishes accountability. How do you hold anyone to account if there is no clear, single set of promises that everyone can see and understand?

In the long run, the constant a/b testing and targeting might even encourage a different type of politician. If politics drifts into a behavioural science of triggers and emotional nudges, it’s reasonable to assume this would most benefit candidates with the least consistent principles, the ones who make the flexible campaign promises. Perhaps the politicians of the future will be those with the fewest ideas and greatest talent for vagueness, because that leaves maximum scope for algorithm-based targeted messaging. What’s really terrifying about all this is not how outrageous it is, but how normal it has already become.

Data from your fridge and your Facebook posts will show you get angrier when you're hungry

Jamie Bartlett’s new book The People Versus Tech is out this April.