We live in a time of political fury and hardening cultural divides. But if there is one thing on which virtually everyone is agreed, it is that the news and information we receive is biased. Every second of every day, someone is complaining about bias, in everything from the latest movie reviews to sports commentary to the BBC’s coverage of Brexit. These complaints and controversies take up a growing share of public discussion.

Much of the outrage that floods social media, occasionally leaking into opinion columns and broadcast interviews, is not simply a reaction to events themselves, but to the way in which they are reported and framed. The “mainstream media” is the principal focal point for this anger. Journalists and broadcasters who purport to be neutral are a constant object of scrutiny and derision, whenever they appear to let their personal views slip. The work of journalists involves an increasing amount of unscripted, real-time discussion, which provides an occasionally troubling window into their thinking.

But this is not simply an anti-journalist sentiment. A similar fury can just as easily descend on a civil servant or independent expert whenever their veneer of neutrality seems to crack, apparently revealing prejudices underneath. Sometimes a report or claim is dismissed as biased or inaccurate for the simple reason that it is unwelcome: to a Brexiter, every bad economic forecast is just another case of the so-called project fear. A sense that the game is rigged now fuels public debate.

This mentality now spans the entire political spectrum and pervades societies around the world. A recent survey found that the majority of people globally believe their society is broken and their economy is rigged. Both the left and the right feel misrepresented and misunderstood by political institutions and the media, but the anger is shared by many in the liberal centre, who believe that populists have gamed the system to harvest more attention than they deserve. Outrage with “mainstream” institutions has become a mass sentiment.

This spirit of indignation was once the natural property of the left, which has long resented the establishment bias of the press. But in the present culture war, the right points to universities, the BBC and civil service as institutions that twist our basic understanding of reality to their own ends. Everyone can point to evidence that justifies their outrage. This arms race in cultural analysis is unwinnable.

This is not as simple as distrust. The appearance of digital platforms, smartphones and the ubiquitous surveillance they enable has ushered in a new public mood that is instinctively suspicious of anyone claiming to describe reality in a fair and objective fashion. It is a mindset that begins with legitimate curiosity about what motivates a given media story, but which ends in a Trumpian refusal to accept any mainstream or official account of the world. We can all probably locate ourselves somewhere on this spectrum, between the curiosity of the engaged citizen and the corrosive cynicism of the climate denier. The question is whether this mentality is doing us any good, either individually or collectively.

Public life has become like a play whose audience is unwilling to suspend disbelief. Any utterance by a public figure can be unpicked in search of its ulterior motive. As cynicism grows, even judges, the supposedly neutral upholders of the law, are publicly accused of personal bias. Once doubt descends on public life, people become increasingly dependent on their own experiences and their own beliefs about how the world really works. One effect of this is that facts no longer seem to matter (the phenomenon misleadingly dubbed “post-truth”). But the crisis of democracy and of truth are one and the same: individuals are increasingly suspicious of the “official” stories they are being told, and expect to witness things for themselves.

On one level, heightened scepticism towards the establishment is a welcome development. A more media-literate and critical citizenry ought to be less easy for the powerful to manipulate. It may even represent a victory for the type of cultural critique pioneered by intellectuals such as Pierre Bourdieu and Stuart Hall in the 1970s and 80s, revealing the injustices embedded in everyday cultural expressions and interactions.

But it is possible to have too much scepticism. How exactly do we distinguish this critical mentality from that of the conspiracy theorist, who is convinced that they alone have seen through the official version of events? Or to turn the question around, how might it be possible to recognise the most flagrant cases of bias in the behaviour of reporters and experts, but nevertheless to accept that what they say is often a reasonable depiction of the world?

It is tempting to blame the internet, populists or foreign trolls for flooding our otherwise rational society with lies. But this underestimates the scale of the technological and philosophical transformations that are under way. The single biggest change in our public sphere is that we now have an unimaginable excess of news and content, where once we had scarcity. Suddenly, the analogue channels and professions we depended on for our knowledge of the world have come to seem partial, slow and dispensable.

And yet, contrary to initial hype surrounding big data, the explosion of information available to us is making it harder, not easier, to achieve consensus on truth. As the quantity of information increases, the need to pick out bite-size pieces of content rises accordingly. In this radically sceptical age, questions of where to look, what to focus on and who to trust are ones that we increasingly seek to answer for ourselves, without the help of intermediaries. This is a liberation of sorts, but it is also at the heart of our deteriorating confidence in public institutions.

The current threat to democracy is often seen to emanate from new forms of propaganda, with the implication that lies are being deliberately fed to a naive and over-emotional public. The simultaneous rise of populist parties and digital platforms has triggered well-known anxieties regarding the fate of truth in democratic societies. Fake news and internet echo chambers are believed to manipulate and ghettoise certain communities, for shadowy ends. Key groups – millennials or the white working-class, say – are accused of being easily persuadable, thanks to their excessive sentimentality.

This diagnosis exaggerates old-fashioned threats while overlooking new phenomena. Over-reliant on analogies to 20th century totalitarianism, it paints the present moment as a moral conflict between truth and lies, with an unthinking public passively consuming the results. But our relationship to information and news is now entirely different: it has become an active and critical one, that is deeply suspicious of the official line. Nowadays, everyone is engaged in spotting and rebutting propaganda of one kind or another, curating our news feeds, attacking the framing of the other side and consciously resisting manipulation. In some ways, we have become too concerned with truth, to the point where we can no longer agree on it. The very institutions that might once have brought controversies to an end are under constant fire for their compromises and biases.

The threat of misinformation and propaganda should not be denied. As the scholars Yochai Benkler, Robert Faris and Hal Roberts have shown in their book Network Propaganda, there is now a self-sustaining information ecosystem on the American right through which conspiracy theories and untruths get recycled, between Breitbart, Fox News, talk radio and social media. Meanwhile, the anti-vaxx movement is becoming a serious public health problem across the world, aided by the online circulation of conspiracy theories and pseudo-science. This is a situation where simple misinformation poses a serious threat to society.

But away from these eye-catching cases, things look less clear-cut. The majority of people in northern Europe still regularly encounter mainstream news and information. Britain is a long way from the US experience, thanks principally to the presence of the BBC, which, for all its faults, still performs a basic function in providing a common informational experience. It is treated as a primary source of news by 60% of people in the UK. Even 42% of Brexit party and Ukip voters get their news from the BBC.

Protesters in London earlier this year. Photograph: Avpics/Alamy

The panic surrounding echo chambers and so-called filter bubbles is largely groundless. If we think of an echo chamber as a sealed environment, which only circulates opinions and facts that are agreeable to its participants, it is a rather implausible phenomenon. Research by the Oxford Internet Institute suggests that just 8% of the UK public are at risk of becoming trapped in such a clique.

Trust in the media is low, but this entrenched scepticism long predates the internet or contemporary populism. From the Sun’s lies about Hillsborough to the BBC’s failure to expose Jimmy Savile as early as they might, to the fevered enthusiasm for the Iraq war that gripped much of Fleet Street, the British public has had plenty of good reasons to distrust journalists. Even so, the number of people in the UK who trust journalists to tell the truth has actually risen slightly since the 1980s.

What, then, has changed? The key thing is that the elites of government and the media have lost their monopoly over the provision of information, but retain their prominence in the public eye. They have become more like celebrities, anti-heroes or figures in a reality TV show. And digital platforms now provide a public space to identify and rake over the flaws, biases and falsehoods of mainstream institutions. The result is an increasingly sceptical citizenry, each seeking to manage their media diet, checking up on individual journalists in order to resist the pernicious influence of the establishment.

There are clear and obvious benefits to this, where it allows hateful and manipulative journalism to be called out. It is reassuring to discover the large swell of public sympathy for the likes of Ben Stokes and Gareth Thomas, and their families, who have been harassed by the tabloids in recent days. But this also generates a mood of outrage, which is far more focused on denouncing bad and biased reporting than with defending the alternative. Across the political spectrum, we are increasingly distracted and enraged by what our adversaries deem important and how they frame it. It is not typically the media’s lies that provoke the greatest fury online, but the discovery that an important event has been ignored or downplayed. While it is true that arguments rage over dodgy facts and figures (concerning climate change or the details of Britain’s trading relations), many of the most bitter controversies of our news cycle concern the framing and weighting of different issues and how they are reported, rather than the facts of what actually happened.

The problem we face is not, then, that certain people are oblivious to the “mainstream media”, or are victims of fake news, but that we are all seeking to see through the veneer of facts and information provided to us by public institutions. Facts and official reports are no longer the end of the story. Such scepticism is healthy and, in many ways, the just deserts of an establishment that has been caught twisting the truth too many times. But political problems arise once we turn against all representations and framings of reality, on the basis that these are compromised and biased – as if some purer, unmediated access to the truth might be possible instead. This is a seductive, but misleading ideal.

Every human culture throughout history has developed ways to record experiences and events, allowing them to endure. From early modern times, liberal societies have developed a wide range of institutions and professions whose work ensures that events do not simply pass without trace or public awareness. Newspapers and broadcasters share reports, photographs and footage of things that have happened in politics, business, society and culture. Court documents and the Hansard parliamentary reports provide records of what has been said in court and in parliament. Systems of accounting, audit and economics help to establish basic facts of what takes place in businesses and markets.

Traditionally, it is through these systems, which are grounded in written testimonies and public statements, that we have learned what is going on in the world. But in the past 20 years, this patchwork of record-keeping has been supplemented and threatened by a radically different system, which is transforming the nature of empirical evidence and memory. One term for this is “big data”, which highlights the exponential growth in the quantity of data that societies create, thanks to digital technologies.

The reason there is so much data today is that more and more of our social lives are mediated digitally. Internet browsers, smartphones, social media platforms, smart cards and every other smart interface record every move we make. Whether or not we are conscious of it, we are constantly leaving traces of our activities, no matter how trivial.

But it is not the escalating quantity of data that constitutes the radical change. Something altogether new has occurred that distinguishes today’s society from previous epochs. In the past, recording devices were principally trained upon events that were already acknowledged as important. Journalists did not just report news, but determined what counted as newsworthy. TV crews turned up at events that were deemed of national significance. The rest of us kept our cameras for noteworthy occasions, such as holidays and parties.

The ubiquity of digital technology has thrown all of this up in the air. Things no longer need to be judged “important” to be captured. Consciously, we photograph events and record experiences regardless of their importance. Unconsciously, we leave a trace of our behaviour every time we swipe a smart card, address Amazon’s Alexa or touch our phone. For the first time in human history, recording now happens by default, and the question of significance is addressed separately.

This shift has prompted an unrealistic set of expectations regarding possibilities for human knowledge. As many of the original evangelists of big data liked to claim, when everything is being recorded, our knowledge of the world no longer needs to be mediated by professionals, experts, institutions and theories. Instead, they argued that the data can simply “speak for itself”. Patterns will emerge, traces will come to light. This holds out the prospect of some purer truth than the one presented to us by professional editors or trained experts. As the Australian surveillance scholar Mark Andrejevic has brilliantly articulated, this is a fantasy of a truth unpolluted by any deliberate human intervention – the ultimate in scientific objectivity.

Andrejevic argues that the rise of this fantasy coincides with growing impatience with the efforts of reporters and experts to frame reality in meaningful ways. He writes that “we might describe the contemporary media moment – and its characteristic attitude of sceptical savviness regarding the contrivance of representation – as one that implicitly embraces the ideal of framelessness”. From this perspective, every controversy can in principle be settled thanks to the vast trove of data – CCTV, records of digital activity and so on – now available to us. Reality in its totality is being recorded, and reporters and officials look dismally compromised by comparison.

One way in which seemingly frameless media has transformed public life over recent years is in the elevation of photography and video as arbiters of truth, as opposed to written testimony or numbers. “Pics or it didn’t happen” is a jokey barb sometimes thrown at social media users when they share some unlikely experience. It is often a single image that seems to capture the truth of an event, only now there are cameras everywhere. No matter how many times it is disproven, the notion that “the camera doesn’t lie” has a peculiar hold over our imaginations. In a society of blanket CCTV and smartphones, there are more cameras than people, and the torrent of data adds to the sense that the truth is somewhere amid the deluge, ignored by mainstream accounts. The central demand of this newly sceptical public is “so show me”.

This transformation in our recording equipment is responsible for much of the outrage directed at those formerly tasked with describing the world. The rise of blanket surveillance technologies has paradoxical effects, raising expectations for objective knowledge to unrealistic levels, and then provoking fury when those in the public eye do not meet them.

On the one hand, data science appears to make the question of objective truth easier to settle. Slow and imperfect institutions of social science and journalism can be circumvented, and we can get directly to reality itself, unpolluted by human bias. Surely, in this age of mass data capture, the truth will become undeniable.

On the other hand, as the quantity of data becomes overwhelming – greater than human intelligence can comprehend – our ability to agree on the nature of reality seems to be declining. Once everything is, in principle, recordable, disputes heat up regarding what counts as significant in the first place. It turns out that the “frames” that journalists and experts use to reduce and organise information are indispensable to its coherence and meaning.

What we are discovering is that, once the limitations on data capture are removed, there are escalating opportunities for conflict over the nature of reality. Every time a mainstream media agency reports the news, they can instantly be met with the retort: but what about this other event, in another time and another place, that you failed to report? What about the bits you left out? What about the other voters in the town you didn’t talk to? When editors judge the relative importance of stories, they now confront a panoply of alternative judgements. Where records are abundant, fights break out over relevance and meaning.

Professional editors have always faced the challenge of reducing long interviews to short consumable chunks and discarding the majority of photos or text. Editing is largely a question of what to throw away. This necessitates value judgements, that readers and audiences once had little option but to trust. Now, however, the question of which image or sentence is truly significant opens irresolvable arguments. One person’s offcut is another person’s revealing nugget.

Political agendas can be pursued this way, including cynical ones aimed at painting one’s opponents in the worst possible light. An absurd or extreme voice can be represented as typical of a political movement (known as “nutpicking”). Taking quotes out of context is one of the most disruptive of online ploys, which provokes far more fury than simple insults. Rather than deploying lies or “fake news”, it messes with the significance of data, taking the fact that someone did say or write something, but violating their intended meaning. No doubt professional journalists have always descended to such tactics from time to time, but now we are all at it, provoking a vicious circle of misrepresentation.

Then consider the status of photography and video. It is not just that photographic evidence can be manipulated to mislead, but that questions will always survive regarding camera angle and context. What happened before or after a camera started rolling? What was outside the shot? These questions provoke suspicion, often with good reason.

The most historic example of such a controversy predates digital media. The Zapruder film, which captured the assassination of John F Kennedy, became the most scrutinised piece of footage in history. The film helped spawn countless conspiracy theories, with individual frames becoming the focus of controversies, with competing theories as to what they reveal. The difficulty of completely squaring any narrative with a photographic image is a philosophical one as much as anything, and the Zapruder film gave a glimpse of the sorts of media disputes that have become endemic now cameras are ubiquitous parts of our social lives and built environments.

Minor gestures pored over for hidden meanings … Emily Maitlis (left) with Nadhim Zahawi and Barry Gardiner on the BBC’s Newsnight. Photograph: BBC

Today, minor gestures that would usually have passed without comment only a decade ago become pored over in search of their hidden message. What did Emily Maitlis mean when she rolled her eyes at Barry Gardiner on Newsnight? What was Jeremy Corbyn mouthing during Prime Minister’s Questions? Who took the photo of Boris Johnson and Carrie Symonds sitting at a garden table in July, and why? This way madness lies.

While we are now able to see evidence for ourselves, we all have conflicting ideas of what bit to attend to, and what it means. The camera may not lie, but that is because it does not speak at all. As we become more fixated on some ultimate gold-standard of objective truth, which exceeds the words of mere journalists or experts, so the number of interpretations applied to the evidence multiplies. As our faith in the idea of undeniable proof deepens, so our frustration with competing framings and official accounts rises. All too often, the charge of “bias” means “that’s not my perspective”. Our screen-based interactions with many institutions have become fuelled by anger that our experiences are not being better recognised, along with a new pleasure at being able to complain about it. As the writer and programmer Paul Ford wrote, back in 2011, “the fundamental question of the web” is: “Why wasn’t I consulted?”

What we are witnessing is a collision between two conflicting ideals of truth: one that depends on trusted intermediaries (journalists and experts), and another that promises the illusion of direct access to reality itself. This has echoes of the populist challenge to liberal democracy, which pits direct expressions of the popular will against parliaments and judges, undermining the very possibility of compromise. The Brexit crisis exemplifies this as well as anything. Liberals and remainers adhere to the long-standing constitutional convention that the public speaks via the institutions of general elections and parliament. Adamant Brexiters believe that the people spoke for themselves in June 2016, and have been thwarted ever since by MPs and civil servants. It is this latter logic that paints suspending parliament as an act of democracy.

This is the tension that many populist leaders exploit. Officials and elected politicians are painted as cynically self-interested, while the “will of the people” is both pure and obvious. Attacks on the mainstream media follow an identical script: the individuals professionally tasked with informing the public, in this case journalists, are biased and fake. It is widely noted that leaders such as Donald Trump, Jair Bolsonaro and Matteo Salvini are enthusiastic users of Twitter, and Boris Johnson has recently begun to use Facebook Live to speak directly to “the people” from Downing Street. Whether it be parliaments or broadcasters, the analogue intermediaries of the public sphere are discredited and circumvented.

What can professional editors and journalists do in response? One response is to shout even louder about their commitment to “truth”, as some American newspapers have profitably done in the face of Trump. But this escalates cultural conflict, and fails to account for how the media and informational landscape has changed in the past 20 years.

What if, instead, we accepted the claim that all reports about the world are simply framings of one kind or another, which cannot but involve political and moral ideas about what counts as important? After all, reality becomes incoherent and overwhelming unless it is simplified and narrated in some way or other. And what if we accepted that journalists, editors and public figures will inevitably let cultural and personal biases slip from time to time? A shrug is often the more appropriate response than a howl. If we abandoned the search for some pure and unbiased truth, where might our critical energies be directed instead?

If we recognise that reporting and editing is always a political act (at least in the sense that it asserts the importance of one story rather than another), then the key question is not whether it is biased, but whether it is independent of financial or political influence. The problem becomes a quasi-constitutional one, of what processes, networks and money determine how data gets turned into news, and how power gets distributed. On this front, the British media is looking worse and worse, with every year that passes.

The relationship between the government and the press has been getting tighter since the 1980s. This is partly thanks to the overweening power of Rupert Murdoch, and the image management that developed in response. Spin doctors such as Alastair Campbell, Andy Coulson, Tom Baldwin, Robbie Gibb and Seumas Milne typically move from the media into party politics, weakening the division between the two.

Then there are those individuals who shift backwards and forwards between senior political positions and the BBC, such as Gibb, Rona Fairhead and James Purnell. The press has taken a very bad turn over recent years, with ex-Chancellor George Osborne becoming editor of the Evening Standard, then the extraordinary recent behaviour of the Daily Telegraph, which seeks to present whatever story or gloss is most supportive of their former star columnist in 10 Downing Street, and rubbishes his opponents. (The Opinion page of the Telegraph website proudly includes a “Best of Boris” section.)

Since the financial crisis of 2008, there have been regular complaints about the revolving door between the financial sector and governmental institutions around the world, most importantly the White House. There has been far less criticism of the similar door that links the media and politics. The exception to this comes from populist leaders, who routinely denounce all “mainstream” democratic and media institutions as a single liberal elite, that acts against the will of the people. One of the reasons they are able to do this is because there is a grain of truth in what they say.

The financial obstacles confronting critical, independent, investigative media are significant. If the Johnson administration takes a more sharply populist turn, the political obstacles could increase, too – Channel 4 is frequently held up as an enemy of Brexit, for example. But let us be clear that an independent, professional media is what we need to defend at the present moment, and abandon the misleading and destructive idea that – thanks to a combination of ubiquitous data capture and personal passions – the truth can be grasped directly, without anyone needing to report it.

• Follow the Long Read on Twitter at @gdnlongread, and sign up to the long read weekly email here.