“All human life is there” used to be the proudest boast of the (mercifully) defunct News of the World. Like everything else in that organ, it wasn’t true: the NoW specialised in randy vicars, chorus girls, Tory spankers, pools winners, C-list celebrities and other minority sports. But there is a medium to which the slogan definitely applies – it’s called the internet.

The best metaphor for the net is to think of it as a mirror held up to human nature. All human life really is there. There’s no ideology, fetish, behaviour, obsession, perversion, eccentricity or fad that doesn’t find expression somewhere online. And while much of what we see reflected back to us is uplifting, banal, intriguing, harmless or fascinating, some of it is truly awful, for the simple reason that human nature is not only infinitely diverse but also sometimes unspeakably cruel.

A key question is whether the moderation task is ultimately a futile, sisyphean one

In the early days of the internet and, later, the web, this didn’t matter so much. But once cyberspace was captured by a few giant platforms, particularly Google, YouTube, Twitter and Facebook, then it became problematic. The business models of these platforms depended on encouraging people to upload content to them in digital torrents. “Broadcast yourself”, remember, was once the motto of YouTube.

And people did – as they slit the throats of hostages in the deserts of Arabia, raped three-year-old girls, shot an old man in the street, firebombed the villages of ethnic minorities or hanged themselves on camera…

All of which posed a problem for the social media brands, which liked to present themselves as facilitators of creativity, connectivity and good clean fun, an image threatened by the tide of crud that was coming at them. So they started employing people to filter and manage it. They were called “moderators” and for a long time they were kept firmly under wraps, so that nobody knew about them.

That cloak of invisibility began to fray as journalists and scholars started to probe this dark underbelly of social media. What has emerged from their investigations is the realisation that content moderation is now a huge global industry employing at least 100,000 people across the world. Most of these are contract workers rather than staff members and they work in stressful conditions, with little job security or health insurance and suffer psychological damage from having to confront – and make decisions about – the horrific stuff that people post on social media platforms.

Sarah T Roberts is an academic at the University of California, Los Angeles, who has been exploring this netherworld of commercial content moderation for eight years. Her book Behind the Screen is the first extensive ethnographic study of those who inhabit it. Nobody who reads it will ever again view social media platforms in a tolerant light. For what it reveals is that the tech industry, like all the great industries of the past, prospers on the back of human exploitation. Our social media feeds are kept clean by the unsung and poorly remunerated labour of countless thousands of human beings who have to risk traumatisation so that we shall not.

In that sense, social media is rather like the fashion business, endlessly providing diversion and pleasure to western consumers by marking up garments that have been made in the sweatshops of India, Bangladesh, Pakistan and Vietnam.

Roberts’s research confirms what scholars such as Tarleton Gillespie had suggested – that far from being a way for social media platforms to demonstrate their sense of social responsibility, content moderation is actually the critical part of their operations, for without it their brands would be irreparably damaged by what unsuspecting users would find in their feeds.

After all, one of the platforms’ pitches to users is that they provide safer spaces than the unruly, uncurated web. As one former moderator replied when Roberts asked him what the platform would be like without moderation: “It would be 100% porn. One hundred per cent. It would be a train wreck. The awful part is, it has to be done. It can’t not be moderated. But there’s no good way to do it.”

A key question, which Roberts doesn’t really explore, is whether the moderation task is ultimately a futile, sisyphean one, given the scale at which the platforms operate. Something like 400 hours of video are uploaded to YouTube every minute, for example. The technocratic response to this challenge is a cheery assurance that, one day, AI will take care of most of it. Roberts and her interviewees don’t buy that and neither do I. And even if Facebook or YouTube employed half-a-million human moderators, it wouldn’t be enough, unless something changes.

One thing that could make a difference would be for the platforms to “throttle” uploads to a more manageable rate. Unsurprisingly, this is an option that is never discussed in the industry. As the computer scientist Hany Farid put it: “The content is just too valuable a commodity to the platforms; it is the bait that lures users in and keeps them coming back for updates to scroll, new pictures or videos to view, new posts to read and new advertisements to be served.”

Corporations such as Google and Facebook that live by surveillance capitalism may therefore be heading for some kind of existential crisis. On the one hand, they have to keep the supply of uploads coming, for they provide the feedstock for their advertising machines; on the other, if through their contractors they cannot manage the moderation task, then their brands will become polluted and users will start to desert their platforms.

But that’s their problem. For us, one of the most striking messages of this remarkable book is that governments and regulators should be investigating the appalling working conditions under which much content moderation is done. And if you’re the kind of person who balks at buying a T-shirt that has been made in an Asian sweatshop, you might think of giving up Facebook or switching to DuckDuckGo for searching. After all, the issues are much the same.

• Behind the Screen is published by Yale University Press (£20). To order a copy go to guardianbookshop.com or call 0330 333 6846. Free UK p&p over £15, online orders only. Phone orders min p&p of £1.99