It's 15 years since Louis Theroux turned the TV cameras on to the US porn industry. Now he is revisiting it to see if anything has changed – and he finds a business in crisis

In a grand suburban house on a quiet cul-de-sac in California's San Fernando Valley an actor is having a problem with her moans. Aleksa Nicole (her professional name) is playing the role of a Latin beauty in A Love Story, a pornographic film about an author of romance novels suffering from writer's block. They are shooting a fantasy sequence in which Aleksa wanders the darkened corridors of the house in a white nightie, carrying a large candlestick. She stumbles into the arms of her forbidden lover, Miguel, played by rising star Xander Corvus, clad in leather trousers, frilly blouse and waistcoat. Helpless in the heat of passion, they make love on the chaise longue.

But there is a small issue. Aleksa's rapid high-pitched squeals of pleasure aren't up to the exacting standards of the film's director.

"Less porno," he says. By way of illustration he offers a different read – less urgent, more ladylike. "Yes, yes, yes!" Then he announces his keyword for the day: "Romantico!"

A Love Story is a new title by the high-end adult movie studio Wicked Picture. And for the world of "adult", the emphasis on the moans is a giveaway that it is not a typical sex film.

For years the porn industry was dominated by an anarchic anything-goes attitude to sex. Directors competed to see who could stage the more outrageous stunts, pushing the performers to the limit of what their bodies could take. The scenes could be hard to watch, as I discovered for myself when I visited sets for a book I was writing in 2004. The sex acts seemed to owe more to reality shows where people eat live worms and pig vomit than anything conventionally erotic.

But some time around 2007, the "business of X" started going into a commercial tailspin. The arrival of free YouTube-style porn sites meant that consumers could download pirated scenes from the vast backlog of old content for free. The phenomenon of DIY amateur sex – part-timers uploading their videos on sites such as clips4sale – also put a dent in the professionals' pay cheques.

Suddenly an industry that was a byword for easy money, raking in billions by exploiting the anonymity of point-and-click purchasing, was fighting for its life.

Making the problems of "adult" even worse was that where consumers might feel enough loyalty to, say, Radiohead to buy their latest release rather than download it illegally, porn users don't have the same feelings about the Dirty Debutantes series. In essence, as with every other media evolution of the last 30 years, from VHS to DVDs to the birth of the internet, porn was once again leading the way, only this time into obsolescence.

And as goes the industry, so go the performers. It's well known that many of them come into porn looking for validation, fleeing lives of damage and abuse. They then sign up to a lifestyle that inflicts stress and illness, not to mention embarrassment, on its young foot soldiers, while offering nothing in the way of pensions and health insurance. Now they find themselves out of work, looking for a Plan B, when the only experience on their resumé is having sex for cash.

On the business side, the porn industry has been desperately trying to adapt. Partly this has been a simple case of cutting back massively. In the early 2000s, a typical issue of the industry bible, the monthly Adult Video News, might have contained hundreds of reviews of new releases. One recent example had just 14. Numerous companies have gone out of business.

Those movie companies that remain are focusing increasingly on high-end product, trying to beat the illegal sites by providing something like a cinematic experience. There is a flight into "quality". In an uncanny echo of a recent BBC slogan, they are embracing the idea of "Fewer, Bigger, Better". For some, this means more female-orientated scenes with less angry sex. Hence A Love Story. For others, it means parodies – of popular TV shows and recent blockbusters.

One of the unlikeliest figures in the new reinvented industry – and a one-man indicator of how much it has changed – is Rob Zicari, better known as Rob Black. In the 90s, Black was one of the most notorious provocateurs in porn. He specialised in tastelessness; his films were more like grotesque exercises in taboo-breaking than anything anyone might conceivably watch for sexual pleasure.

In 1997 I interviewed him in his office in LA and visited him on the set of a production entitled Forced Entry, a film about rape. He was only 23 at the time and I was struck by the strange contrast of his being a friendly, intelligent guy – albeit in an over-caffeinated way – while making porn films that specialised in degrading women. Six years later, Black's provocations caught up with him during George Bush's "war on obscenity" (the war's two other casualties were Justin Timberlake and Janet Jackson, when she exposed her nipple in a dance routine during the Superbowl).

Black and his wife and business partner Janet Romano (better know as Lizzie Borden) did a year each in federal jail for obscenity. Post-prison, a chastened Black has put his Sadean grotesqueries behind him. Now he directs superhero parodies for a mainstream porn company called Vivid – Captain America XXX, Iron Man XXX – in glossy two-disc sets, some of them in 3D.

"Where the business is going now is it's acceptable to sit down with your wife and girlfriend and introduce her to pornography," he tells me during a visit to his new offices. "But the stuff you're going to introduce them to is the stuff I'm making."

He shows me the suit he had used for his Iron Man parody. I remark that in porn terms, it is an impressive prop. "Dude, the suit cost like four grand!" he says. Then, picking up one of his CD cases, in mock-awe at its resemblance to a "real movie", he says: "Look at that! That's a porno!"

Black is adept at putting a positive spin on the retrenchment that porn had undergone. But he appears somewhat ravaged and looks older than his 38 years. He has the air of someone who has been through something that hasn't killed him, but which hasn't made him stronger either.

If times are hard for the Rob Blacks of porn, they're worse still for the men on screen. Even with the superhero parodies and the couples films, and the lower-paid work doing scenes for pay-sites, there is still nowhere near enough employment for the hordes of performers who hope to make a living getting paid to have sex on camera.

At one of the top LA agencies for performers, LA Direct, the accountant Francine Amidor laments the "devastating" impact of piracy. "There's less work, and there's an abundance – because of the economy – of performers. There aren't enough people shooting to give everybody a day's work."

I put it to Amidor that she owes it to the young aspirants who still make their way to the LA Direct offices to explain the consequences of their decision. She demurs. "Because then I would talk three quarters of the girls out of the business and then we wouldn't be in business."

Fees for scenes, not surprisingly, have taken a hit. "Some girls get $600 [£390] for a scene now," the retired performer JJ Michaels tells me. "It might be $900-$1,000 for a big-name girl. It used to get up to $3,000." For guys, rates can be $150 or lower.

Women supplement their income by stripping and doing live shows over the internet, shot from home on their webcams. One evening I visit one of LA Direct's top performers, Kagney Linn Karter – star of Rack-Tastic and Pound the Round – at her house as she prepares for her bi-monthly live show. Her boyfriend and full-time assistant Montae is hanging up her dresses while Kagney bathes and puts on her makeup. Montae and I then retreat to the kitchen where he tidies and wipes down surfaces while Kagney strips on her bed and masturbates in front of the strangers viewing her through her laptop. Forty-five minutes later, she emerges. "Well, I made a hundred dollars," she says brightly.

It's an open secret in the porn world that many female performers are supplementing their income by "hooking on the side". It's also called "doing privates", as in private bookings. The official industry line is that it's dangerous (because clients aren't tested the way performers are) and irresponsible (because the women could then infect the closed community of professional performers). But the women can make far more money having sex behind closed doors than doing it on film and, in fact, the practice is widespread. For many female performers nowadays, the movies are merely a sideline, a kind of advertising for their real business of prostitution.

Male performers do not have the same options. For a tiny subsection of top talent, there is still a regular pay cheque, albeit a shrinking one. But work has dried up for many of the journeyman-performers in the lower ranks and there is a great deal of anxiety across the board.

In the 90s one of the best-known male performers was Jon Dough – birth name Chet Anuszak. He was on contract with Vivid – the only man in the business to be exclusive to a company at that time. He had a reputation for being a dependable "woodsman" and was well liked in adult circles. I interviewed him in 1996 on the set of a remake of Debbie Does Dallas. But when he killed himself in 2006 at the age of 43 it was widely assumed that the woes of the industry – and specifically declining DVD sales – played a role.

Jon Dough's widow is a fellow performer whose stage name is Monique DeMoan. They met on the set of Dr Butts 3: The Anal Asylum and eventually had a daughter together. No one in the world of porn knows Monique's whereabouts but I tracked her down to a conservative state many hundreds of miles – physically, culturally – from Los Angeles.

Standing in the low-ceilinged basement flat in an insalubrious area, she says that her husband killed himself over his cocaine addiction, and the instability and sense of failure that went with it, not because of the pressures of the industry. Still, the perception among Dough's peers that DVD sales were a factor reflects an emotional truth: people in the world of porn were all too ready to believe that a top performer might have killed himself over the decline of a media format.

Still they arrive, the cohorts of aspiring performers, looking for new lives of wealth and stardom in a world that can no longer offer either. If Paris Hilton and Kim Kardashian can make sex tapes, they ask, then why shouldn't they? They tour the production companies for meet-and-greets with casting agents and directors, tick boxes on questionnaires about what they are willing to do on camera. They make one or two low-end scenes for the internet, before being chucked back into anonymity.

On a rainy day, back on the same spacious suburban house on the quiet street in the valley where A Love Story was filmed, another production is under way. This time, it is a more modest affair – a few scenes made for an internet site. One of the male performers is a young man on his second shoot, performing under the name Tony Prince.

His partner for the scene is called Stefania. It is her third shoot and she seems excited to have press around. "I'm trying to become one of the big porn stars," she says. She asks to take a picture with me, which she later tweets.

The scene is supposed to look like a real-life boyfriend and girlfriend sex tape. One advantage of this is that there is no need for a cameraman. The performers shoot the action themselves. While they got down to business, to alleviate any possible performance anxiety on Tony's part, the director and I make ourselves scarce in the kitchen. He has been in the business since 1998, and he too is pessimistic about its future. "It's like I tell guys all the time, you better make this your side gig," he says.

A few hours later, the two performers are laughing and showering together, both happy with how the shoot had gone. In a reversal of the usual order of things, they are flirting and becoming friends after sex.

Where the industry will end up is hard to predict. Clearly there is still a market for softcore movies made by companies such as Penthouse and Hustler, available on subscription channels. The parodies may continue for a while, too. But it is difficult to see how a business selling hardcore movies and even internet clips is sustainable when most people simply don't want to pay if they don't have to. To many people, when it comes to porn, not paying for content seems the more moral thing to do.

"The way it is now, within five years I don't see how there could be a professional porn actor," Michaels tells me. It's not easy to sympathise with the porn companies, which made so much money for so long by embracing a tawdry business and a dysfunctional work-pool. But it is worth sparing a thought for the legions of performers, qualified for nothing much more than having sex on camera, who have no money saved, and no future.

And there is also the wider question: do those who use porn not, perhaps, owe it a little something? Should those who download it not be ready to pass on a little cash incentive to the business? And if not, why not? Does the stigma attached to porn make it OK to steal it? These questions underpin a much bigger dilemma being faced by all media: how do you sustain an industry that provides a certain standard of product – be it journalism, music, or mainstream movies, or X-rated movies – when more and more consumers are in the habit of downloading content for free? In the world of porn, the answer is: you can't.

• Twilight of the Pornstars is on BBC2 on Sunday at 10pm

• This article was amended on 15 June 2012 because the original version misnamed some of the films and people it mentioned. The correct names are: Pound the Round, not Pound Round; Rack-Tastic, not Racktastic; Aleksa Nicole, not Alexa Nicole; and Montae, not Monte.