By early 2017 the world had woken up to a problem that, with a mixture of impotence, incomprehension and dread, journalists had seen coming for some time. News, the thing that helped people understand their world, that oiled the wheels of society, that pollinated communities, that kept the powerful honest – news was broken.

The problem had many different names and diagnoses. Some thought we were drowning in too much news; others feared we were in danger of becoming newsless. Some believed we had too much free news; others, that paid-for news was leaving behind it a long caravan of ignorance.

On this most people could agree: we were now up to our necks in a seething, ever churning ocean of information, some of it true, much of it wrong. There was too much false news, not enough reliable news. There might soon be entire communities without news, or without news they could trust.

How did we get here? And how could we get back to where we once belonged?

For 20 years I edited a newspaper in the throes of this tumultuous revolution. The paper I took over in 1995 was composed of words printed on newsprint involving technologies that had changed little since Victorian times. It was, in many ways, a vertically arranged world. We – the organs of information – owned printing presses and, with them, the exclusive power to hand down the news we had gathered. The readers handed up the money – and so did advertisers, who had few other ways of reaching our audience. But today advertisers can often reach consumers much more effectively through other channels. People are much more reluctant to part with money for news. And, however you measure it, there is widespread scepticism, confusion and mistrust about mainstream journalism.

A few years after I became editor of the Guardian, I was invited to give an after-dinner speech at the Thirty Club – a private gathering of big commercial cheeses in the advertising and media worlds. It was March 2003 – three years before the launch of Twitter and the creation of the Facebook “news feed”, before the collapse of the economic model that had underpinned journalism for a century.

The subject of my speech was trust – and the truly abysmal ratings newspapers had in that department. Depending on the poll and the year, we were lucky if 13%-18% of the population trusted newspapers.

As I stood up to speak, I was very conscious of the contingent from News International across the table. Les Hinton, the amiable if faintly menacing executive chairman of Rupert Murdoch’s newspaper group, was sitting right opposite me, flanked by Rebekah Brooks, editor of the Sun, and Andy Coulson, editor of the News of the World. I tried to avoid catching my colleagues’ eyes as I noted that the tabloids sold most, but were trusted least.

I likened British journalists to fans of Millwall football club, who famously chant “No one likes us, we don’t care”. I knew what Coulson and Brooks were thinking: we’re in for a pious sermon from someone who can barely make a profit and whose sales are embarrassingly small. I could hear the ritual jibe from Piers Morgan, then editor of the Mirror, every time he saw me: “I sell more copies in Cornwall than you do in the entire country.”

I waffled on more about trust; how we’d lost it; how to earn it back; why it would matter so much more in the digital world. It was, even if true, worthy stuff. Afterwards, the three Murdoch colleagues were very friendly. They suggested we go on to a club; we ended up drinking in Soho House till the early hours. The champagne’s on them. The speech is not mentioned. The evening is fun. Brooks and Coulson are good company. Hinton is full of seen-it-all bonhomie. Deep down, we’re all hacks together.

Cut to 11 years later: Coulson was in jail, Hinton had resigned and Brooks had suffered the ordeal of a nerve-shredding trial at the Old Bailey – all because of reporting in the Guardian. That night in Soho House now feels like a lost world of Fleet Street innocence. A funny word – “innocence” – to use about Fleet Street. But we were certainly all innocent of what was to come – in virtually every way possible.

In 1976, when I began my career as a journalist, the common route into the business was being thrown into the newsroom of a local paper to learn on the job. A week after finishing my finals, I swapped my university college – founded in 1428 – for the prosaic 1960s offices of the Cambridge Evening News, a mile to the east. There were not many graduates in the 20-strong reporting room of the CEN, a paper then selling just fewer than 50,000 copies a day. University types were – rightly – viewed with suspicion as arrogant interlopers who would trade the experience we gained in the provinces to secure a better-paid job in Fleet Street.

The paper was owned by one Lord Iliffe of Yattendon, a largely absent figure who owned a 9,000-acre estate 100 miles away in Berkshire. More important to me was Fulton Gillespie, the chief reporter, known as Jock – a growling, silver-haired Glaswegian with dark glasses and the stub of a cigar permanently lodged between bearded lips.

Jock saw it as his duty to school us in hard knocks. We would begin the day with the calls – a round trip to the police, ambulance and fire services. As we set off in the office Mini, he would deliver one of a small repertoire of homilies about our craft. “If you write for dukes, only dukes will understand, but if you write for the dustman, both will understand. Keep it short, keep it simple, write it in language you would use if you were telling your mum or dad.”

He explained that police work involved keeping one foot on the pavement and one in the gutter. You got their respect by kicking them in the balls at regular intervals, because, in the long run, they needed us more than we needed them. That, he emphasised, was a good rule applicable to all those in authority. It had been hammered into him by the old hacks on the Falkirk Herald, and it would always be true. He repeated this often in case I had failed to grasp it: they needed us more than we needed them. We owned the printing presses; they didn’t. End of.

Rusbridger in 1982. Photograph: Garry Weaser/The Guardian

Around the same time, I learned a more personal lesson in the ways and obsessions of the press. While I was a young reporter on the CEN, I fell in love. The relationship lasted just under two years. I was a cub reporter, she was a university lecturer. Nobodies. The relationship caused some happiness, and some unhappiness to a few people – literally, no more than half a dozen either way. End of story. Well, almost – her late father had, some years earlier, been on the telly. So you could, at a stretch, make a consumable tale out of it: “Daughter of quite famous man has affair.”

One Friday night, there was a knock on the door. A reporter and photographer from the Sunday Mirror wanted to tell the “story of our love”, as he put it, to the 4 million readers who then bought the newspaper every week. The reporter, a man called Richard, was charming, but when we politely declined the opportunity to invite him in, Richard’s tone changed. “We can do this nice or we can do it nasty,” he said abruptly, and then explained what nice and nasty looked like. Nice was for us to sit down on the sofa and tell the world about our love, and be portrayed in a sympathetic way. Nasty meant they would start knocking on the doors of neighbours and contacting our relatives to put together a story that would be altogether less heartwarming.

It was a good pitch. All the same, we felt this was, well, private. We were living together openly, and made no attempt to hide our relationship from friends or family. But we had no wish to tell the whole world. So we said no.

Richard and his photographer sat outside the house for another 24 hours. From time to time he would lean on the doorbell to test whether we had changed our minds. A week later, the pair reappeared to try again. Eventually we asked them in for a cup of tea, and I suggested I might ring Richard’s news editor to explain that we wouldn’t be talking. That seemed to do the trick. The story – nice or nasty – never saw the light of day.

My life at that point had been learning to report councils, courts, freak weather and flower shows. That was what I understood journalism to be – a record of public events of varying degrees of significance. The ring on the doorbell was my first, sharp realisation that journalism meant many different things to many different people. And there were many different business models for “journalism”.

I joined the Guardian in 1979, on the same day as Nick Davies, who went on to become one of the most fearless and prolific investigative reporters of his generation. Our paths intersected for 35 years – his as a reporter, mine as an editor. When he came to see me in 2005 to propose that he would close his career by writing one last big series about power – specifically, the unchecked power of the press – I knew it would lead somewhere difficult. Nick was a heat-seeking missile. If he spent the next year digging, you could guarantee he would come up with something extraordinary. He always did.

The germ of the idea had come from the Iraq war and the press’s role in aiding and abetting a conflict based on what now we would call fake news. But the elders of Fleet Street did not appreciate the diligent attention of investigative journalism when it was turned upon them. On the Guardian we had long published a Monday media section – and, latterly, a media website. Over time, I was threatened by not quite every publisher or editor in London, but a good many of them.

Editor 1: “I’ll always retaliate. We should be sticking together, not writing about each other.”

Editor 2: “My circulation is three times yours. You write about me, I’ll write about you. In the end you’ll stop.”

Editor 3: “I hate writing about the media, but I make an exception for the Guardian.”

Publisher 1: “We’ve both got ink wells, Alan. Remember that.”

Publisher 2: “I’ll release the hounds of hell on you. You have to stop writing about us.”

Some tried for bilateral deals: “You don’t write about us, we won’t write about you.” In general, they were as good as their word. If the Guardian upset a rival editor or publisher, you could guarantee a form of retribution within days. It could be a snide diary paragraph or a threat to dig up dirt on Guardian employees’ private lives – including mine. It could be a hatchet job about “troubles” at the paper. Sometimes these stories had a grain of truth, at other times they were simply invented. At first I was shocked that even broadsheet rivals would knowingly run untrue stories about us in “revenge”. After a while I understood that we were all supposed to see this as the great game.

I had no idea what Nick would discover, but nor did it feel right that the Fleet Street elders should get their wish that their journalism should be a coverage-free zone. If journalism is a force of immense influence – and I think it is, and should be – then it surely deserves scrutiny.

Investigative journalism of this sort is very slow, expensive, and sometimes yields very little direct return. No management consultant on earth would conclude that it represents a sensible investment of time or resources: a newsroom run strictly on metrics could never justify it. Big investigations (Watergate, for example) often work via the incremental disclosures of sometimes not very headline-grabbing material. The readership for each of the smaller numerous stories along the way would be barely measurable. The great editor Harold Evans used to say that an investigation only really began to count once the readers – and even the journalists – were bored with it.

So how to justify obsessive reporting that has no apparent financial rationale, and may even be of little interest to the readers? The answer to this question is central to the idea of a newspaper. If journalism is, in some sense, a public service, then an editor has to understand the ethos of public service – something that is of value to a society without necessarily making a direct financial return. This means thinking of this kind of journalism in the same way you might think of a police, ambulance or fire service. You would, as a citizen, expect such services to be run efficiently, but you would not expect them to have to justify themselves on grounds of profit.

There is actually a financial benefit to investigations, but it is a long-term one. Readers, on some level, want their newspapers to be brave, serious, campaigning and dogged. They like corruption to be exposed, overweening power to be challenged, and serious scandals to be unearthed. It reminds them what journalism is for. They admire it. They are even willing to pay for it.

A newspaper that consistently breaks investigative stories will (with apologies to those who hate the word) build a brand. The Harold Evans Sunday Times was certainly a “brand”. To this day, it is regarded as one of the high-water marks of challenging 20th-century journalism. Brands have value. A paper that stands for nothing will soon lose its sheen, and then its point, and then its readers. But that’s not always an immediately winning argument if the financials are looking tense and you have impatient investors.

Davies’s quest to explore the power of the press had resulted in a fiercely forensic book, Flat Earth News, which chronicled how many newsrooms, obsessed with traffic and with ever-declining budgets, had started practising what he termed “churnalism”. The word stuck, because so many editorial staff labouring under the pressures of shrinking resources and ever-faster output recognised the truth of it. But Nick’s inquiries, it turned out, were only just beginning. He came to see me in March 2009 to tell me about a shocking new story. We would later refer to this as the “heart-attack conversation”.

A News of the World reporter, Clive Goodman, had been jailed in 2007, along with the private eye who had helped him, for intercepting the voicemail of three people who worked at Buckingham Palace. The editor, Andy Coulson, had resigned – but he was now the communications director for the Conservative party, and would soon be on his way to Downing Street when David Cameron became prime minister in 2010. The official line from News International was that Goodman had been a rotten apple: his phone-hacking was a one-off.

Davies told me this story was not true. He had been contacted by a source who told him the idea that Goodman was the only person to hack phones was a joke. Loads of reporters were at it: it was how the News of the World had won so many awards. Hacking phones was the system, not an aberration.

The police knew this at the time, but had done nothing about it. But now one of the other victims of hacking – Gordon Taylor, the chief executive of the Professional Footballers’ Association – was suing, and was trying to find out who had known what, and when they knew it.

How many victims were we talking about? Nick had met a senior figure at Scotland Yard. The answer: thousands. So, not just one rotten apple then.

The News of the World, rattled by this new legal action, had offered to pay Taylor an enormous sum – £400,000 plus £300,000 costs – to drop the action. Together with the payments to Taylor’s colleagues, News International was offering to pay no less than £1m to make the actions go away.

Nick had been told that hacking victims included the deputy prime minister, John Prescott. Dozens of News of the World reporters and executives were implicated. Nick had access to emails showing that transcripts of 35 voice messages had been discussed by named reporters and editors. The “rogue reporter” defence was shot to pieces.

According to Nick’s sources, the deal had been approved by James Murdoch, son of Rupert and chair of News International. The silence money had been paid, and the court documents sealed. If Nick was right, Murdoch’s most senior UK executive had agreed to a million-pound cover-up of criminal behaviour in his own company.

From left: Les Hinton, Andy Coulson, Rupert Murdoch and Rebekah Brooks, in 2005. Photograph: Graeme Robertson/The Guardian

This was an incendiary story. The Murdoch operation, taken as a whole, was ruthless. If we merely wounded the company, it would close in for the kill. We already knew that the police, for reasons best known to themselves, would not want to get involved. We would not have many friends in politics or the rest of the press. We would be on our own.

Our story ran mid-afternoon on Wednesday 8 July 2009. It detailed the conspiracy to cover up criminal behaviour. It implicated the Conservative leader’s spokesman. It accused Murdoch’s executives of misleading parliament. It pointed a finger of blame at the press regulator, and it asked why the police had turned a blind eye.

Fleet Street showed only mild interest in the story. News International had released an official three-page statement rubbishing our work and exonerating themselves. The company’s PR operation had been working overtime in Westminster. All the allegations were, they said, false. Rebekah Brooks wrote to the chairman of the select committee saying we had deliberately misled the British public. The Times took a piece from another former Scotland Yard officer (now employed by Murdoch) pouring cold water on the Guardian story. It was reprinted in the sister paper, the News of the World, under a full-page editorial attacking us.

It was a lesson in how the Murdoch organisation fought back. A senior executive on the Sun later promised to use the pages of the Sunday Times to show that I was the “biggest fucking hypocrite in the world”. It was as if the family of titles were interchangeable in being used to target anyone with the temerity to take on the organisation. The entire Murdoch UK newspaper organisation appeared to have been mobilised to call the truth fake; and to promote fake news as the truth.

Over the next two years, our reporting was gradually and painfully vindicated. On 15 July 2011, Brooks resigned from News International. Two days after that, she was arrested. She was eventually acquitted, but Murdoch’s appearance before MPs – “the most humble day of my life” – showed an organisation in moral and organisational turmoil.

Those two years it had taken to fight off the Murdoch press’s fury and prove that Nick had been right were lonely ones. You live in a democracy, you assume that there are numerous checks and balances to prevent powerful people from doing crooked things. For the first time in my adult life, I doubted this was true in Britain.

We had presented strong evidence of a criminal conspiracy at one of the most powerful media companies in the world – and no one wanted to know. Not the police. Not the regulator. Not – initially, at least – parliament. And not the British press.

The British gatekeepers eventually found themselves on trial – or, at least, subjected to the unforgiving glare of a judicial inquiry – with the Leveson inquiry into the culture, practice and ethics of the British press, the government’s response to Nick Davies’s reporting, which opened on 14 November 2011.

The two-part inquiry had been set up by the prime minister, David Cameron, in the wake of the phone-hacking revelations. Until this point, the British press financed its own self-regulatory body, the Press Complaints Commission. The British press had defended this organisation as tough and rigorous – right up to July 2011, when the elders had admitted it was pretty toothless after all, and should be replaced by something tougher and more rigorous.

But there was a case to answer. Unchecked criminality within newsrooms was a moral catastrophe for British journalism and its role in our democracy. It was our Enron, our Volkswagen, our Deepwater, our subprime crisis. It was depressing to watch some colleagues retreat into the bunker and use their own bully-pages to close down debate by savaging anyone who offered even constructive help in rebuilding trust in the press. I loathed the threats and abuse directed at anyone who dared to disagree.

The initial response to Leveson was, in fact, measured. There seemed, if anything, a sense of relief that we had escaped relatively lightly. But some of the aftermath was badly handled all round, and it did not take long for Fleet Street to circle the wagons – bitterly rejecting even the best-intentioned attempts to craft the most credible form of independent regulation.

The dripping contempt for “the liberal elite” or their supposed notion of “public interest” was the bitter stuff of deeper culture wars. But Leveson and its fallout clarified how much confusion – or, more accurately, straight disagreement – now existed about the nature and purpose of what we did.

It was becoming more and more evident that there was no neat agreement on what journalism in the public interest looked like. The Daily Mail employed many outstanding reporters, but the relentless, bruising, sometimes brutalising editorial ethos of that paper had little in common with the BBC or the Financial Times, any more than Fox News had much in common with the New York Times or Washington Post.

Even among the so-called legacy media organisations, there was a yawning divergence in attitudes to what journalism should or could be. Scarcely a week passed without a withering attack from the Mail on the BBC’s editorial ethos or standards. Periodically, the Mail editor would castigate the Guardian, Channel 4 or the FT – either personally, anonymously or through willing surrogates. There was never a flicker of irony or humility in the eyes of the most complained-about newspaper (1,214 PCC complaints in 2013) bellowing contempt at the most trusted news organisations in the country (the FT had just seven PCC complaints in 2013).

The elders’ fury at Leveson was dressed up in invocations of Cobbett or Swift. But there seemed little that Leveson himself was then proposing – as opposed to the botched subsequent wrangling – that would inhibit robust commentary or investigation into matters of genuine public significance. What, then, did they fear? The real fury – also apparent in denunciations of the European court’s jurisdiction concerning privacy – was reserved for judges who stood in the way of the lucrative pursuit of sex scandals.

The tabloids have a business model that, in part, involves intrusion into the privacy of people in the public eye. In comparison to some other business models, it’s not a bad one. The public’s appetite for scandal subsidises the coverage of politics. Take away the sleaze and the sex and you can kiss goodbye to reportage of Westminster – that’s how they argue it.

But it was hard not to wonder at all the puffed-up outrage. Was the Sun editorial team itself more free of “filthy laundry” or “sleazy antics” than any other newsroom? Journalists – and even some proprietors – were not known for leading saintly lives, any more than footballers.

Did it matter? The Sun and Mail can surely have their economic imperative and ethics; the New York Times and FT can have theirs. Fox News and the BBC: it’s all “journalism”. We can rub along in peaceful coexistence. That’s how it used to work, in a kinder age before newspapers began using the techniques of the secret police to spy on their targets.

But now journalism is facing an existential economic threat in the form of a tumultuous recalibration of our place in the world. And on both sides of an increasingly scratchy debate about media, politics and democracy, there is a hesitancy about whether there is still a common idea of what journalism is, and why it matters.

On 3 December 2013, I was led into a parliamentary committee room in Westminster to justify journalism. It was not a trial, but it felt like one. Waiting in the very next room were two of the most senior police officers in the country, who were investigating whether I should be prosecuted. Some Conservative members of the committee were determined to force admissions out of me and would, I felt sure, quite like to see me in jail.

The Home Affairs committee was notionally looking into counter-terrorism. But today they wanted to know about Edward Snowden, the former US National Security Agency (NSA) operative who had given the Guardian and others a very large number of top-secret NSA and GCHQ documents revealing the extent of state surveillance activities. Such a leak had never happened before.

The head of MI6, the urbane former diplomat Sir John Sawers, had told MPs that Britain’s adversaries were “rubbing their hands with glee. Al-Qaeda is lapping it up.” He had smoothly delivered the next day’s headlines.

The British press, in contrast to the overwhelming majority of journalists in virtually every other country, had not exactly bent over backwards to support the Guardian in publishing the Snowden revelations. Some had been openly hostile. A former broadsheet editor had written that newspapers had no right to determine the public interest when it came to security. An Economist writer had suggested that he would have marched Snowden to the police if he had brought him the story. Once again, the British press could not agree what the public interest looked like.

The MPs arranged in front of me in a horseshoe did not, on the whole, look friendly. I was on my own. The committee chair, the maverick Labour MP Keith Vaz, opened the questions. He came from a Goan family and had settled in England at the age of six after a period in Aden. We had not been going for long when he lobbed what felt like a fizzing grenade in my direction. “You and I were both born outside this country,” he said. “But I love this country. Do you love this country?”

For a split second I was speechless. I recovered to say that my patriotism was rooted in the idea of a Britain that allowed a free press that could report on such matters. There were countries where the security services told editors what they could or couldn’t write. They weren’t democracies. I was proud to live in a country that didn’t behave like that.

If journalists cannot agree on a common idea of the public interest – of the public service we claim to be providing – then it complicates the defence of what we do. And in an age of horizontal free mass media, it is even more important for us to be able to define and declare our values, our purpose – and our independence. Which includes independence from the state.

But five years after the Snowden revelations, it is now apparent that states themselves are struggling with the digital disruption that first tore through the established media and has now reshaped politics. The digital giants have not only unleashed information chaos – they have, in the blink of an eye, become arguably the most powerful organisations the world has ever seen.

‘Public” is, to some in the 21st century, a difficult word. We value public services, public spaces and public goods – but we sometimes struggle to know how to discuss them, create them, run them, fund them, regulate them, support them or measure them. We speak of public benefits and “the public interest” without ever satisfactorily defining them. In the UK, we treasure a public health service – but not to the point of financing it sufficiently. We trust a public service broadcaster above all private news providers – but regularly revile it.

The ultimate defence of journalism is that it remains a public good. But how do we measure, or value, such a public good at a time when, in the words of the political philosopher Michael Sandel, “markets – and market values – have come to govern our lives as never before … Markets leave their mark. Sometimes, market values crowd out non-market values worth caring about.”

A decade ago, talking this way about news would have been dismissed as the bleating of liberals – the despised “subsidariat” who weren’t commercial enough to produce decent journalism that the public wanted to read. (As James Murdoch infamously claimed in 2009, “The only reliable durable and perpetual guarantor of independence is profit.”) But now, faced with the overwhelming scale, popularity and sheer turbo-charged commercial power of the Silicon Valley titans, there is now an outcry across the media against the havoc that free markets have inflicted on the traditional press.

After two decades of disruption, it may be possible that none of the old conventional business models can still support serious news in the public interest. But the challenge has never been so urgent: we need the essential work of journalism – the calling that should, at its highest, separate lies from the truth.

This is an edited extract from Breaking News: The Remaking of Journalism and Why It Matters Now, published by Canongate on 6 September, and available at guardianbookshop.com

Alan Rusbridger will be in conversation with Beeban Kidron at the Barbican in London on 5 September. For tickets click here

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.