It was only a couple of decades ago, before WiFi or the static death shriek of dial-up modems, that fetishes were hard-won secrets. The porno most of us stumbled across in our formative pubescence – a Playboy left on a rural Greyhound bus, or a waterlogged bodice-ripper in a back-alley dumpster – was all hard silicon protuberances and breathy nonsense. It was sexual, so it was almost invariably scintillating. But it was also ultimately just a flattening manifestation of one grotesque, misogynistic fantasy. Figuring out what actually got your motor running required work and guts. You had to dig deep into the bargain boxes of porn shops, open up with strangers (or worse, those you loved), and pray that you could find someone who understood and would accommodate your desires.

The internet changed all of that. Especially as search engines matured, digital voyeurs learned that they could spurt their every impulse and urge into the search bar. And on the other side of the enter key, they would almost invariably find forums collectively celebrating individuals’ secret desires, or enterprising smut-mongers catering directly to them. ‘It had a naming effect,’ says Michael Stabile, a renowned gay pornographer. Today he’s a spokesperson for the website Kink, a pioneer in internet fetish porn that has become a Mecca for folks with an interest in BDSM (bondage, dominance and submission, and sadomasochism). ‘Sexuality has always operated in tandem with pornography. Pornography tends to crystallise desires that you might not have articulated.’

Humanity hasn’t yet worked out how it feels about the internet’s effect on our collective sexuality. Some see it as a liberating force, opening us up to new levels of pleasure, satisfaction and self-understanding. Others see it as corrosive: transgressive stuff like diaper porn – in which young girls stripped naked save for a pair of Depends roll on the floor cooing and gurgling – stokes the flames of paedophilia, prevents people from finding joy in vanilla sex, and drives them to more extreme and troubling perversions, they say. Many more just plug up their ears to avoid this whole uncomfortable and often confoundingly euphemistic debate.

Yet while we debate the ethics of naïfs in nappies, most of us have missed a vital new chapter unfolding in this ongoing saga: the internet is on the verge of another raunchy revolution, one that could again shift the contours of human sexuality. Whereas for the past two decades, erotic entrepreneurs have poured smarm into static fleshpots scattered around the interwebs, confident that fetishists would come to them, now the kink is coming to us. These days you can boot up Pornhub, xHamster or any other popular porn tube site that collects videos from around the web, and there’s a decent chance that you’ll see a moving thumbnail of a topless girl in a diaper. You’re also likely to see flashing images of ‘fauxcest’ porn, in which actors pretending to be step- or blood relations go to town on each other; ‘futa’, in which women ‘grow’ dicks and fuck each other; or some other fetish you used to have to scour to the dark edges of the net to find.

That transsexual gangbang you fapped to last night? That high-def rosebud you clicked on? They probably didn’t just float to the top of some indiscriminate data dump to reach your attention. You saw them because producers and distributors are fitfully learning to harness big data. The desire stew that you dumped into a site’s search bar: they probably recorded and learned from it. Data on what we search for, pay for and click on is being used to predict our desires and funnel us bespoke(ish) porn.

At first blush, it might seem like this kind of micro-targeting would just turbo-boost the internet’s existing trajectory, making it even easier for people to find and embrace a diversity of bodies and fetishes. But there’s a fundamental shift here from a world in which we explore a passive sea of content to a world in which porn actively explores and prescribes itself to us. Because this shift stems from deep financial upheaval in the adult industry, the content pushed upon us will likely increasingly reflect what is most profitable, not what is most widely desirable. It could well become narrowing, or at least channelling, rather than broadening.

Nothing about this nascent shift is certain. But there’s a chance that the intersection of big data, big business and ubiquitous connectivity will give market forces a new foothold to dictate our sexual development via an avenue that, for many, long felt private, idiosyncratic and liberating. In the end, capitalism might prove itself the ultimate arbiter of desire.

Porn is a data goldmine. Going by Alexa traffic rankings, the porn tubes XVideos, Pornhub and xHamster are, right now, the 51st, 63rd, and 88th most-visited sites in the world, respectively. Sites such as these get billions of page views per month from tens of millions of people, all of whom leave (thanks to tracking cookies and IP data) little traces of themselves behind. Studio producers alone churn out more than 10,000 skin flicks per year catering to these horny hordes, far outpacing Hollywood’s 500-or-so offerings. By some (albeit sketchy) estimates, porn adds up to 4 per cent of all sites, 14 per cent of all searches and 30 per cent of all data transfers online.

That pot of numbers is itself porn for the global datarati – the folks who love nothing more than dissecting humanity by the numbers and restructuring the world to streamline our impulses in the name of efficiency. Yet according to Alec Helmy, publisher of the adult industry rag XBIZ: ‘While the importance of data analysis has reached a fever pitch in mainstream tech, it is not yet a major priority for much of pornography.’ In a world dripping with information, porn has remained one of the most intuitive, unscientific and misunderstood modern industries.

The prudish aversion to data’s gaze might seem bizarre, since the porn industry has historically positioned itself as an early adapter of and leader in tech trends. In 1897, just two years after the first commercial screening of a film, bawdy Victorian-era bros started setting sex onto celluloid; the industry pioneered VHS distribution in the 1980s, allowing a copy of Debbie Does Dallas (1978) to find its way into every sock drawer; and perfected most of the tools (and banes) of e-commerce in the 1990s, from electronic billing to encryption, making it relatively safe and easy to stream even the raunchiest images on to your desktop in the dead of night.

Around the turn of the millennium, pornographers even started to use cookie trackers and in-site member surveys to figure out what videos to make within their niche markets. Husband-and-wife pornographers Angie and Colin Rowntree have relied on survey data to create most of the scenes on their for-women site Sssh for the past 17 years. Angie developed the film Gone (2015) based on the account of a Sssh member who’d lost her significant other, a collaboration directly responding to requests for story-driven porn. Gone earned mainstream media buzz for its heat and artistry, and attracted the duo even more for-the-plot viewers – not to mention a deal of scratch.

Porn culture didn’t want big data, and porn economics didn’t need big data – for a while, at least

But according to Ogi Ogas, a Harvard visiting scholar and co-author, with the data analyst Sai Gaddam, of A Billion Wicked Thoughts: What the Internet Tells Us About Sexual Relationships (2011), at its heart the adult industry until fairly recently remained insular and intuitive. As the wider world transformed, porn producers continued to indulge their own at-most-lightly-mediated impulses and gut instincts. They created a bit of everything, filled every possible opening from spit roasting to double penetration to double vaginal. As long as there was a stable market for every type of content, the competitive pushes and pulls that set other industries in search of datagasms couldn’t make serious inroads into porn.

Just as important, anyone inclined to jump on the big-data wagon in the industry wound up frustrated by its fragmentation. Although there was a ton of data out there on the intimate desires of the world, it was spread across thousands of sites, most of which were aggressively opaque, maintaining an air of discretion for customers eager to keep this one slice of their lives private. Porn culture didn’t want big data, and porn economics didn’t need big data – for a while, at least.

Data ultimately got its hooks into porn because of YouTube. By 2007, two years after the video-sharing service launched, erotic entrepreneurs had created clones such as RedTube, Youjizz, and YouPorn. Ostensibly created to make it easier to upload and share free amateur videos, a lack of oversight quickly turned them into repositories for millions of brazenly pirated professionally made films, too.

Denizens of the web flocked to these tube sites, attracted by their troves of diverse and free content. When Ogas investigated them in 2008, they weren’t yet thinking much about data collection. But they soon caught on to the fact that their huge content base and high traffic had inadvertently created the cohesive datasets that the porn industry had always lacked. Powered by ad revenue, the tube sites felt an economic tug to draw more people to their sites for longer chunks of time. Since they had – at least initially – no control over content creation, the name of the game on tube sites for the past few years was content funnelling.

Almost every tube site now has complex tagging systems to optimise searches or to recommend videos based on your viewing habits, ‘trending video’ pages, and some kind of program to push certain content via those channels that’ll draw and lock in desired users. Few sites are willing to open up about the mechanisms by which diaper porn occasionally percolates to the top of the heap for some unsuspecting wankers. But sources at xHamster, the third-largest tube site out there, were willing to tell me about the basics of their content-pushing, which is probably similar to that of other sites.

Every day, a team at xHamster watches roughly 2,500 newly added pornos. Looking at user data collected via Google Analytics and proprietary in-house tools, they determine which ones to put on the front page of the site. They also tweak and develop algorithms to recommend the perfect video based on your searches and watching habits. They’re constantly running A/B testing on their users – throwing different front pages, for instance, to learn more about how your demographic reacts to anal prolapse or Czech gangbang content.

One of the easiest pieces of data to mine is your location, so sites such as xHamster place special focus on learning what porn is hottest where you live, and feeding you more of the same. Everyone loves watching girls dance on webcams, a representative of xHamster told me, but beyond that every country gets something special. ‘German people don’t like the interracial content,’ the rep says. ‘And the Japanese are extremely patriotic. They have zero interest in other categories but for national Japanese ones.’

Colin Rowntree of Sssh, a self-avowed big-data skeptic, says he can get behind this practice. ‘If a site knows that surfers from Iceland tend to prefer tall, muscular blondes wearing furry boots, why not offer those viewers just that?’ he muses. ‘This is the old tried and true Netflix “getting-to-know-you” technology, which is very useful for enhancing the user experience.’

There’s nothing wrong with promoting a fetish. But it’s likely to make certain proclivities more visible

Regional tracking is superficially similar to what the Rowntrees were already doing with their surveys: learning more about their customers and using that information to give them more of the content they’re looking for. But porn-tube data is much broader and blunter than survey results. Given the number of cautious people who search for porn only after disabling cookie trackers and logging out of their other accounts, or while using proxy IP addresses that mask their location, and given the flaws in often crowd-sourced video tags, it’s entirely possible that tubes are messing up. I could be an American kid living in Saudi Arabia and using a German proxy to get my jollies, and as a result never get exposed to the interracial porn I might love. Customised content could be configuring my impressionable mind towards a whole different set of sexual inclinations, submitting it to a haphazard tyranny of the masses – which may not even by my masses.

Even when sites get their targeting right, they might still have a financial interest in drawing in a demographic that is not you. If Pornhub is targeting more women (as they say they are), they might try pushing more story-heavy movies – which today often means fauxcest. In the process, they might wind up pushing incest role-play videos on users who would have never sought them out, driving up interest and engagement in that particular fetish. There’s nothing prima facie wrong with promoting a fetish. But it’s likely to happen disproportionately for some fetishes over others, making certain proclivities more visible than others, and thereby altering our sexual reference points, all in the name of ad clicks.

Jack Kona, a director with 15 years’ experience in the adult industry, thinks that most old-school producers are suffering from Donald Trump Syndrome: they act like they’re rolling in the dough, but when pressed they’re incredibly cagey about their finances. The media, from highbrow think pieces to TV shows such as Silicon Valley, is quick to perpetuate these claims. American porn alone is often pegged as a $50 billion industry.

It’s not. Free tube sites and their rampant piracy have eviscerated traditional pornography. From 2007 to 2011, the industry collapsed by 50 per cent; Kink, an especially successful site, reported its first losses at the end of that period. Larry Flynt, the notorious Hustler mogul, only half-jokingly requested a federal bailout for the industry in 2009. Today, Thierry Arrondo, the managing director of Vendo, an electronic pricing and billing platform serving a number of porn sites, estimates that paysites collectively make about $500 million a year, while live webcam sites pull in another $1 billion.

‘No adult online content provider is going to go belly-up showing young women having sex,’ Ogas notes. But Kona says he’s working four times as hard to make half the money he used to. You basically can’t make any money out of straight, vanilla sex anymore, because there’s so damn much of it available for free. To goose their profits, porn producers – much like distributors – have started to turn to data mining. Unlike the tube sites, though, they’re not using it passively. Instead they’re using it to guide the kind of content they create, deliberately shifting the industry toward more extreme, more profitable genres.

Most porn sites don’t have as much raw data as the tube sites do. So, Arrando explains, they play around with new content and see what does best. They try to offer something others don’t, whether that’s fisting every orifice possible in one sitting or scripting the largest ever faux-incestuous gangbang. For straight sex scenes, performers are doing more physically taxing things – such as anal prolapse, which requires a medical procedure right afterwards every time – more often, and often for much less money. For fetish scenes, straight intercourse is now laced with fake necrophilia, paedophilia or dramatised rape. That’s a big part of why fauxcest, which is much easier to film and perform than, say, a double penetration gangbang, has become incredibly popular, growing by up to 1,000 per cent over the past five years by one estimate.

Some producers just spray content on the wall to see what sticks. But some operations are starting to pay attention to which videos are trending on tube sites; at the same time, sites such as Clips4Sale, a platform for à-la-carte sales by small producers, collect deep and easily accessible data on consumer desire and pricing and payment trends, which producers can freely and easily crunch. The result is cascades of content, with studios and amateurs alike riffing on the same marketable fetishes, giving us years where you suddenly can’t escape cuckold porn or jerk-off instructions.

That herd behaviour is logical, but it’s also troubling for your average schmuck. It means that producers no longer have as much freedom to produce an ocean of diverse content. Instead they churn out a disproportionate amount of content aimed at sexual niche groups that’ll pay more than others. According to Odette Delacroix, an actress and fetish producer who runs a popular diaper site, she branched into that taboo only after she ran her numbers and realised how much folks were willing to pay for that content. And so the cycle unfolded, probably like so: Delacroix made more diaper porn; some of it ended up on the tube sites; devotees found the free content and ate it up; tube sites saw the traffic numbers and threw a clip or two onto their trending page. In the end, some casual fappers wound up with an eyeful of Delacroix as a dirty baby front and centre on their one-stop porn page.

a few decades back, anal was a relatively rare fetish. Today, it’s a routine element of everyday Americans’ sex lives

This push to produce more extreme content in waves, catering to the highest paying customers’ tastes, could soon grow even stronger as tube sites branch out of distribution and start creating their own videos. One firm especially, MindGeek, currently operates or owns 13 tube sites, 10 production companies, and has ties to Playboy and other porn studios such as Really Useful Ltd and Wicked Pictures. ‘Since MindGeek runs most leading [tube] sites, and owns a range of key US production companies, they have a unique position,’ says Susanna Paasonen, a pornography expert at Finland’s University of Turku. They have better data than their competitors on a wider range of people, allowing them to zero in on precise gaps in the market. They could notice, for example, that there’s an intersection between the market for Asian BBW (big beautiful women) porn and porn of girls wearing sailor outfits (to riff on a facetious hypothetical Ogas presented to me). Then they could test to see what happens when they make a movie featuring BBW Asians in sailor uniforms.

We like to think that we’re totally capable of separating pornographic sex from real sex. But, a few decades back, anal was a relatively rare fetish. Today, it’s the bread and butter of mainstream porn, and a routine element of everyday Americans’ sex lives. The same thing happened with oral a few decades prior. Not everyone is convinced that pornography alone drives these changes in lived sexual norms, but it’s definitely at least a contributing factor – and cause for enough concern that folks see fit to make digital TV channels and public service announcements to teach kids the difference between porn and real sex.

If MindGeek starts to produce its own niche content, it’ll heap even more pressure on competitor producers to hold on to the most profitable corners of the market. Overall, the porn that’s out there will get more extreme. Things that were once fetishes might become routine. Young consumers could find themselves growing more comfortable with sexual gymnastics and other behaviours long regarded as oddities. Human sexuality might be increasingly shaped by what kind of porn will still sell in an era beset by filthy, filthy pirates.

It might seem peculiar that there has been so little cultural objection to the spread of porn that actively targets its customers, that makes increasingly potent recommendations, and could preferentially guide viewers toward extreme fetishes. My hunch is that part of the reason for our blasé acceptance stems from the sanitised way that the porn tubes have presented their use of data. In 2013, Pornhub formed a user-data analysis unit and immediately made marketing and social outreach the team’s public priority, according to their vice president Corey Price. Its first ever post that June tracked traffic on the site during the National Basketball Association finals – a gimmick designed to court BuzzFeed-style coverage and to treat porn consumption as just one more bit of pop-culture gossip.

Meanwhile, behavioural experts vocally doubt whether targeted pornography will really affect us, given how rarely people change their established preferences, to which they flock regardless of what’s being flung at them. The rise of porn data, some of them argue, just makes it easier for people to locate and talk about their sexual happy places. Antoine Mazières, one of the brains behind Sexualitics, a blog that mines porn data for insights on human behaviour, suggests that even that user-experience boost is a long way off, given how poorly some tube sites’ tagging systems operate. Industry experts echo this attitude; Kink’s Stabile thinks that targeting will at most introduce people to fetishes they’re already predisposed to like but wouldn’t have known how to find in the past.

Yet these perspectives discount the experience of novices diving in with ill-formed preferences and of those who just tend to go with the flow. They also do not reflect the reality of an adult industry whose economic foundation has been thoroughly shattered. The pursuit of the most rabidly loyal audience has already driven political websites and talk radio to polarising extremes, with vast societal implications. Fearing similar outcomes in the world of pornography and sexuality is not entirely outlandish.

Big data and big dollars are working their way into our popular conception of sexuality – with the public yielding to the desires of business, rather than the other way around. How sophisticated that slow creep becomes and to what extent it manages to penetrate our lived realities remains unclear. I personally doubt we’ll see a world where we all start to incorporate double-fisting into our sex lives. But I have almost no doubt that we’re headed to a world where novice porn consumers load up SpankBang for the first time, see Delacroix rolling around in diapers, and think: ‘So this is what people like me like.’