Cambridge Analytica may have become the byword for a scandal, but it’s not entirely clear that anyone knows exactly what that scandal is. It’s more like toxic word association: “Facebook”, “data”, “harvested”, “weaponised”, “Trump” and, in this country, most controversially, “Brexit”.

It was a media firestorm that’s yet to be extinguished, a year on from whistleblower Christopher Wylie’s revelations in the Observer and the New York Times about how the company acquired the personal data of tens of millions of Facebook users in order to target them in political campaigns.

This week sees the release of The Great Hack, a Netflix documentary that is the first feature-length attempt to gather all the strands of the affair into some sort of narrative – though it is one contested even by those appearing in the film.

“This is not about one company,” Julian Wheatland, the ex-chief operating officer of Cambridge Analytica, claims at one point. “This technology is going on unabated and will continue to go on unabated.[…] There was always going to be a Cambridge Analytica. It just sucks to me that it’s Cambridge Analytica.”

Cambridge Analytica didn’t decide democracy was for sale. We built this world, so we should own it

It’s true that, since the scandal broke, we have discovered that Facebook was leaking data all over the place over many years. But, in the film, Wheatland conveniently glosses over Cambridge Analytica’s more singular attributes, such as offering “electoral services” that included entrapment using Ukrainian prostitutes and admitting to bribing officials in Caribbean elections. But having worked for more than a year to bring Wylie forward, it’s fascinating for me now to glimpse how the scandal continued to unfold.

Wheatland describes what it was like to be inside a news tsunami: at its height, he says, they were dealing with an almost fantastical 35,000 news stories a day.

The Great Hack is the work of Jehane Noujaim and Karim Amer, the husband-and-wife team who made The Square, the Oscar-nominated film about the Arab spring. Here, they tell the story via the personal journeys of two contrasting individuals: David Carroll, a New York media professor who attempts a circuitous, difficult and ultimately unsuccessful journey via the English legal system to find out what data Cambridge Analytica held on him; and Brittany Kaiser, an ex-employee of Cambridge Analytica who turned “whistleblower”.

Carroll’s doomed attempt to lift the veil from the data-industrial complex that underpinned Cambridge Analytica is the dark heart of the film. Because although he proved that the firm had illegally processed his data, ultimately his attempt to retrieve that data was thwarted by Cambridge Analytica’s decision to liquidate.

Carroll’s experience is just one of the many unknowns that still surround this story. We still know very little about what the company actually did with the data. Who was targeted? With what ads? In what locations? Carroll knows nothing about the nature of the 5,000 data points the firm claimed, in its own marketing, to have on 230 million American voters, including himself. We still have no clear picture what Cambridge Analytica did for Trump. Or what it did in any of the dozens of elections worldwide it claimed to have worked on – what Carroll calls “subversion on an industrial scale”. All we know is that both Cambridge Analytica and Facebook have gone to extraordinary lengths to prevent the facts coming out.

The data swamp remains dark, toxic and invisible. But what the film tries to do through creative and unusual graphics is to make the invisible visible: pixels representing data bytes float off Carroll as he rides the subway – the informational exhaust fumes we give off, hundreds of thousands of data points every day, which are hoovered up and monetised by the tech monopoly giants in ways we can’t see or understand.

Timeline Cambridge Analytica Show First hint of the scandal The Guardian reports that political data firm Cambridge Analytica was helping Ted Cruz's presidential campaign, suggesting the Republican candidate was using psychological data based on research into tens of millions of Facebook users in an attempt to gain an advantage over his political rivals Firm works with Trump campaign

Cambridge Analytica, the political consultancy firm of which Steve Bannon is vice-president, starts working with Trump campaign aide Brad Parscale in San Antonio, alongside employees from Facebook and Google. Two months later, Donald Trump sacks Paul Manafort as his campaign manager and appoints Bannon. The campaign spends $6m on Cambridge Analytica’s services The first legal action David Carroll, an American professor, files a case to reclaim his data from Cambridge Analytica under English law. Wylie reveals harvest Christopher Wylie, a co-founder of Cambridge Analytica, claims in the Observer that the firm used 50 million harvested Facebook profiles in a major data scandal. This number was later revised by Facebook to 87 million. Wylie claimed the data sold to Cambridge Analytica was then used to develop "psychographic" profiles of people and deliver pro-Trump material to them online Channel 4's revelations Channel 4 broadcasts its undercover films of Cambridge Analytica’s CEO, Alexander Nix, discussing his role in Trump’s 2016 election. Zuckerberg apologises After four days of refusing to comment, Mark Zuckerberg publishes a Facebook post apologising for the data breach. The Facebook CEO responds to the continued fallout over the data scandal, saying: "We have a responsibility to protect your data, and if we can't then we don't deserve to serve you. I've been working to understand exactly what happened and how to make sure this doesn't happen again" The ‘sorry’ ads Zuckerberg takes out full-page ads in a number of British and American newspapers to apologise for a "breach of trust"

Kaiser testifies Ex-Cambridge Analytica director Brittany Kaiser testifies before the digital, culture, media and sport select committee. Facebook revenues rise Facebook releases its first earnings report since the scandal was reported. The quarterly revenue was its highest for a first quarter and the second highest overall Investigations launched Cambridge Analytica goes into administration on 3 May. Days later it is reported that the FBI and the US Justice Department are investigating the company. On 16 May, Wylie appears before the US congress to answer questions about the scandal. y Nix testifies Following the liquidation of Cambridge Analytica, Alexander Nix finally appears before the DCMS select committee. Facebook fined The UK's Information Commissioner’s Office announces it intends to fine Facebook £500,000 ($663,000) over the data scandal, saying Facebook "contravened the law by failing to safeguard people's information".

$119bn is knocked off Facebook's stock value when Zuckerberg announces that significant numbers of users are leaving the platform ‘Empty chair’ Having refused multiple invitations to appear before the UK parliamentary inquiry into fake news, Mark Zuckerberg is "empty chaired" at a special committee meeting of members of nine national parliaments DCMS report published After Zuckerberg refuses to testify, the DCMS report into fake news is finalised – and concludes that the UK's electoral laws are not fit for purpose in the digital age. Criminal inquiry It is reported that the US Justice Department is conducting a criminal inquiry into Facebook's data-sharing with other technology companies High court ruling The high court rules against appointing new administrators of Cambridge Analytica – thwarting Carroll’s efforts to retrieve his data.

Exposing this invisible world of manipulation and power is one of the principal aims of the film, according to Amer and Noujaim. But it springs from the unlikeliest of places: the techno-optimism of the Arab spring, the revolutions that were widely greeted as proving the liberating effects of this new technology. The couple met in Egypt’s Tahrir Square during the months-long protest in 2011. Noujaim was filming and Amer was organising, but they ended up working together on the film that became The Square.

“We saw first-hand how this technology was this incredible tool for change,” says Noujaim. “I was arrested, and it was a human rights lawyer tweeting my picture that became the only way they could find me.”

This led the couple to try to find a story about technology, but gradually it morphed into a very different film.

“We started out looking at the Sony hack [the cyberattack on emails at Sony Pictures in 2014], but soon felt that the hack that was most interesting was not the physical hack but the brain one: the manipulation that was happening to our minds.”

The Observer front page, Sunday 18 March 2018.

The project mutated, moving its focus first to the US election and then to the trillion-dollar-a-year data industry, and took so long that they had three children (including twins) while filming it, across three continents. In the summer of 2017 they approached me to be a “character”, an offer I declined (although a year later I relented and did a couple of interviews). More important, I introduced them to experts I’d found on Twitter: David Carroll, associate professor of media design at Parsons School of Design in New York; Paul-Olivier Dehaye, a data rights expert in Switzerland who had helped me research and understand Cambridge Analytica, and Ravi Naik, Carroll’s (and my) London-based human rights lawyer, who believes data rights represent the next great human rights battle.

The film chronicles Carroll’s attempt to assert his rights to his data under UK law, and for him, it was about telling the parts of the story that the US press almost entirely failed to grasp.

“This was a military contractor that got into the business of election management. That alone is so unsettling. Everyone got so fixated on the Facebook data. My case proved that what Cambridge Analytica was doing with US voters’ data was illegal [under English law]. The story is about so much more than misappropriated Facebook data. In fact, that’s the least troublesome part of the grand narrative that The Great Hack puts forward.”

There is so much that is troublesome, including, Carroll points out, the other central figure in the film, the Cambridge Analytica “whistleblower” Brittany Kaiser. Whistleblowers are often complex and contradictory, and Kaiser, a former senior director at the company, comes across as elusive and evasive.

Amer says that he met Kaiser for the first time, as we do in the film, in a pool in Thailand in bikini and sunglasses, her eyes hidden. We never find out why she’s in the pool, or in Thailand, for that matter. There is no mention of the fact that she decided to blow the whistle the day after the Guardian published a story on her involvement in an allegedly criminal smear campaign in the Nigerian 2015 election that was extreme even by Cambridge Analytica’s standards.

David Carroll in a scene from The Great Hack. Photograph: Netflix

The ambiguity of Kaiser’s involvement “echoes our own complicity”, Amer says, “because we all played a part in this. It’s not like it’s Cambridge Analytica’s fault that the entire democratic process has been commoditised and you can buy and sell all this data. It’s not Cambridge Analytica’s fault that the US election is a multi-billion-dollar market. Cambridge Analytica didn’t decide democracy was for sale, we allowed that. We built this world, so we should own it… and not just point fingers at people who exploited it.”

It’s a neat argument, but some are perhaps more complicit than others. Noujaim and Amer treat Kaiser in good faith and call her story a “redemption narrative”. But to others – including Wylie, Cambridge Analytica’s former research director, and Carroll – she’s an unreliable narrator. The narrative, Carroll says, is one that has netted her a “high six-figure advance” for a biography coming out later this year. “I hope people can see the difference between those who are monetising the Cambridge Analytica story and those who are doing the work.”

And the film’s omissions of parts of Kaiser’s story are a shame. I spent months tracking down ex-employees who claimed their lives were put at risk after Kaiser brought in a team of Israeli intelligence agents for the Nigerian election campaign who hacked the current president’s emails. Employees describe how they were directed to search through these to find damaging material to leak to the press. There were “uncanny” echoes, employees said, of what happened with WikiLeaks and Hillary Clinton’s emails in the US election 18 months later.

People have completely misunderstood the scandal as being about privacy, when it’s actually about power

There’s no mention of that in the film, though. There are moments when Carroll expresses scepticism about Kaiser, and, most dramatically – just as the CEO of Cambridge Analytica, Alexander Nix, is being questioned by a select committee of British MPs about why he contacted Julian Assange to ask him for Clinton’s emails – we learn that Kaiser had her own contact with WikiLeaks. The viewer finds out about this because I wrote a story about it – in June 2018, with Guardian journalist Stephanie Kirchgaessner – and in the film we see Kaiser reacting to news of the story in real time. “Carole knows I visited Julian Assange in February,” she says tearfully, after reading a message on her phone. “And she knows I gave money to WikiLeaks in bitcoin.” Later, she reveals she has been subpoenaed by US justice department special counsel Robert Mueller and is seen travelling to give evidence. (Her lawyer later said she had “discharged her obligations”.)

Kaiser has brought forward important evidence, but has also been reluctant to acknowledge her own complicated role. She was asked by MPs about her relationship with WikiLeaks and omitted to mention her visit to Assange.

“Don’t take this the wrong way, but in your life have you worked for or provided information to any country’s intelligence agencies?” Ian Lucas, Labour MP for Wrexham, asks her at the parliamentary hearing. The film captures the moment of her response, a single alarming eyebrow that shoots up.

According to Shoshana Zuboff, professor emerita at Harvard Business School, the Cambridge Analytica scandal was a landmark moment, because it revealed a micro version “of the larger phenomenon that is surveillance capitalism”.

Zuboff is responsible for formulating the concept of surveillance capitalism, and published a magisterial, indispensible book with that title soon after the scandal broke. In the book, Zuboff creates a framework and a language for understanding this new world. She believes The Great Hack is an important landmark in terms of public understanding, and that Noujaim and Amer capture “what living under the conditions of surveillance capitalism means. That every action is being repurposed as raw material for behavioural data. And that these data are being lifted from our lives in ways that are systematically engineered to be invisible. And therefore we can never resist.”

Brittany Kaiser, former director of Cambridge Analytica, at a screening of The Great Hack at Sundance in January. Photograph: David Becker/Getty Images for Netflix

She, too, is troubled by the multiple questions that the scandal raised and that remain unanswered. And she’s exasperated by the people who scoff at the idea that what Cambridge Analytica did was some sort of “dark arts”. The whole point is that we just don’t know what really happened, she says. There has been no forensic analysis. “And it’s not because the data doesn’t exist. It does. It’s just that Facebook is sitting on it. That’s why we need the law. What they want is for this to be so obfuscated, so difficult to get to the bottom of it, that we just let it fade into this kind of haze of ‘Well, it seemed really bad, but we just don’t know… maybe it was overblown.’ And doubt fills the void.”

For Zuboff, Cambridge Analytica has demonstrated how it’s possible for any motivated billionaire to leverage tools to build his own scaled-down version of the massive surveillance and manipulation machines that are Google and Facebook.

Last week, Facebook was fined $5bn for its part in the scandal by the US Federal Trade Commission, one of dozens of ongoing investigations into the company around the world. Facebook is being scrutinised, if not actually held to account ($5bn has no meaningful impact on a company of its size and scale). But Cambridge Analytica’s central role in the US election and its contested role in Brexit remain a mystery.

Martin Moore, an academic at King’s College London and the author of an excellent book on the wider subject, Democracy Hacked, suggests that this is because Cambridge Analytica’s key players have come to be seen as “pantomime villains”. The undercover Channel 4 film aired in the same week as the Observer revelations, he suggests, gave people the licence to dismiss them. “The message that people took away from that was that there’s no substance to what they were doing at all,” says Moore.

Cambridge Analytica’s former CEO Alexander Nix arrives at a Commons select committee, 6 June 2018. Photograph: Tolga Akmen/AFP/Getty Images

“There was this bizarre situation where journalists were falling over themselves to say that this digital media was really overhyped, that it didn’t really make any difference, when we have no idea what was done, and it’s incredibly premature to say we can dismiss all this stuff. We know that Brad Parscale [Trump’s digital director] was putting out 50-60,000 ads per day [during the 2016 US presidential election] and using AI on those ads. And we know that AI has moved on significantly since then.”

Even as the US remains in the dark about what happened in 2016, Trump is accelerating towards the next election; Parscale has revealed a war chest of $1bn and is spending $1m a week on Facebook ads.

Cambridge Analytica is dead, but it’s the living dead. Zombie sons of the company live on, including Data Propria, which is now working on the Trump2020 campaign and, most directly, in Emerdata, a company that was set up by Rebekah and Jennifer Mercer, the daughters of Cambridge Analytica’s original owner, Robert Mercer, alongside Alexander Nix and – later – Julian Wheatland (though both men have since resigned).

It’s extraordinary that, despite the headlines and the articles and the congressional and parliamentary hearings, there is so much we still don’t know about what Cambridge Analytica actually did. We know now that we’re guinea pigs in a vast global, online experiment. We just have no idea what that means.

The film is the first major creative project to grapple with the subject in all its complexity. David Carroll’s single-minded mission and Brittany Kaiser’s erratic transatlantic meanderings provide a personal frame to these larger unanswered questions.

Q&A What did the ICO uncover when it investigated Cambridge Analytica? Show The Cambridge Analytica files resulted in a multi-year investigation from the UK Information Commissioner's Office, "the most important ever", according to Elizabeth Denham, the Information Commissioner. In an interim report, published in July 2018, the ICO announced a number of enforcement actions against a whole spectrum of political actors, including: • A £500,000 fine against Facebook for two breaches of the Data Protection act. • Warning letters to all 11 political parties with MPs in the House of Commons, and notices compelling them to agree to audits of their data protection practices. • An enforcement notice for Cambridge Analytica's parent SCL Elections to compel it to deal properly with a subject access request from US Professor David Carroll, whose data it held, which has already led to a criminal prosecution for for failing to properly deal the notice. • An enforcement notice for Canadian electoral services firm Aggregate IQ, ordering it to stop processing retained data belonging to UK citizens which was provided by Vote Leave. • A notice of intent to take regulatory action against data broker Emma’s Diary, which sold data about new mothers to political parties that had been collected in a "concerning" fashion. • Audits of the main credit reference companies. • An audit of of the Cambridge University Psychometric Centre, where Alexander Kogan worked

One of the few slivers of insight we have into how Cambridge Analytica and Facebook worked together on the Trump campaign comes from a clip used in The Great Hack from a BBC documentary, in which journalist Jamie Bartlett interviews a member of Trump’s digital team who shows where employees of the two companies worked alongside one another.

Parscale told a conference recently that the day the Facebook employees arrived was like getting a “Christmas present”. He recalled the time he told the Trump campaign that Facebook was “the 500 pound gorilla that’s ready for picking. Who controls Facebook controls the 2016 election.” Yet only a fraction of the 5.9m ads that were pumped out by the campaign have ever been seen by the wider public. We know nothing of the modelling that was used, or of the three voter suppression operations that a campaign official told a journalist ahead of the election that they were running. Or almost anything at all. It is all still wrapped in darkness.

Paul-Olivier Dehaye goes as far as to suggest that, in investigating links between Trump and Russia, we are looking in the wrong place for evidence of “collusion”, and describes a scenario that is “more subtle, delicate and dangerous” than any of those presently understood. Collusion, he says, “may have been enabled by Facebook”, its algorithms exploited by “adversarial machine learning”. A malevolent actor, he says, could for example create fake profiles of Hillary supporters, and then use those profiles to train Facebook’s algorithm in a way that would benefit Trump’s campaign.

We know Facebook co-operated with Robert Mueller’s inquiry, but there’s no hint in the redacted report of what he found. Just as we don’t know when Facebook’s executives first learned about Cambridge Analytica’s activities. More than a year on, that most basic question – who at Facebook knew what, when? – has simply not been answered.

Shoshana Zuboff, who coined the term ‘surveillance capitalism’. Photograph: PR

“We saw what happened to Facebook’s share price in the wake of the scandal,” says Jason Kint, the CEO of Digital Content Next, who has been an assiduous observer of the ongoing investigations. “If executives had information that they didn’t tell shareholders, that’s a very big deal.” Facebook has said it is co-operating with investigations, which it takes seriously.

Although the Cambridge Analytica scandal is a lot about Facebook, it doesn’t mean Cambridge Analytica can be let off the hook. It doesn’t make the questions about its activities any less troubling.

There’s a gobsmacking moment, captured in the film, when Kaiser tells the select committee that the British government had classified Cambridge Analytica’s technology as “weapons-grade”. It was export‑controlled, meaning the company was bound by law to tell the government if it used it overseas. Damian Collins, chair of the committee, is seen digesting this piece of information, and asking: “So what you are saying is that the proposal was for Leave.EU to use what you call ‘weapons grade communications techniques’ against the UK population?”

It’s worth reading that sentence again. There’s no conspiracy here, just important unanswered questions. Did the British government have oversight of Cambridge Analytica’s activities in multiple elections worldwide? Was this “weapons grade” technology used on the British population?

And it’s worth restating what kicked off the entire story. It was only when Cambridge Analytica started writing repeatedly to me, furiously denying any involvement in the EU referendum – despite having previously boasted about it in multiple places online – that I started investigating this byzantine web of people, data and money.

The official position, including that of the Electoral Commission, is that Cambridge Analytica did no work for Leave.EU. However, that is not the impression left by the film. Collins says, “We’ve documented how Cambridge Analytica worked with Leave.EU, regardless of whether they were paid or not. There are still so many unanswered questions that require further investigation.

Chris Wylie (right) at a Fair Vote rally in London, 29 March 2018, alongside Shahmir Sanni, who blew the whistle on funding irregularities in the Vote Leave campaign. Photograph: Neil Hall/EPA

“The film does a brilliant job of telling the story of the central role data plays in politics, and the way a few big companies can get inside the heads of millions of people.”

Will we ever know the truth, I ask Zuboff. “If today’s lawmakers and researchers can’t answer these questions, they’re going to ring through history,” she says.

Dehaye describes it as an “abysmal” state of affairs that “we cannot even establish basic facts about potential illegalities by five different actors – Cambridge Analytica, [Alex] Kogan [the academic who mined the Facebook data], the platform, the politicians and the Russians. That’s five different actors involved in using this technology. And we don’t have access to even the most basic information about what they did.”

Still, he’s upbeat when I speak to him just after he’s watched the film. “I feel pretty good,” he says. “In December 2016, I reached out to 10 Americans like David [Carroll] with this idea for an experiment [in which] they would ask for their data through European courts. And now it’s a Netflix movie. So that feels good, even if I don’t have any money or income right now.”

Carroll risked his own money, too. Because nobody pays for any of this, it’s been a citizens’ fight – academics and activists, alongside journalists, backbench MPs and film-makers. The little guy up against the big guy, where the big guys have all the money in the world – the technology behind this has created the wealthiest and most powerful companies on Earth.

Protests over Mark Zuckerberg’s refusal to attend parliamentary hearings into Facebook’s data abuse, November 2018. Photograph: Jack Taylor/Getty Images

“People have completely misunderstood the scandal as being about privacy,” says Naik, “when it’s actually about power.”

And the idea that people are dismissing any of this because they think it doesn’t “work perfectly” is a terrifying proposition, Zuboff believes. “These are monopolies pouring the might of capital into a population-scale experiment and then figuring out how to make it work better and better. This is an infinite curve that has infinite resources and that’s the road we’re on.”

After a near five-year struggle to make their film, and to highlight what they call “the wreckage sites” of surveillance capitalism, Karim Amer and Jehane Noujaim believe that the fightback is in full swing. “It was an amazing moment for us when we met you all, these sleuths who’d found each other online and were using these different ways of hacking the system back.”

Most days, it’s hard to see it that way: because most days, the companies are winning. “The story revealed this huge gaping wound in Silicon Valley where we could see what this beast was,” says Naik. “But we ended up just peering over the edge.”

We’re still there now. Suffering vertigo. Or confusion. The complexity of this story and the ability of those involved to spin it – corporate lobbyist Nick Clegg’s recent round of interviews on behalf of Facebook is just one example – is one of the greatest barriers to people understanding it. For Zuboff, the Cambridge Analytica scandal was the beginning of the great awakening. And The Great Hack is a crucial start of the next stage: “Because now we need action,” she says. “We need law. We have to solve this.”

The Great Hack is available on Netflix and showing at the ICA, London SW1, from 24 July