June 1999, and the music industry is rolling in cash. Bloated by the vast profits of the CD business, which reached a high of £30.6bn that year, the future looks bright for the suits at the top. Sadly for them, in a dorm at Boston’s Northeastern university, a precocious coder is about to blow it all apart.

Napster, the brainchild of Shawn Fanning, linked computers and allowed users to access each other’s mp3 audio files. Fanning met an ambitious teen, Sean Parker, on the webchat channel w00w00, and together they triggered a series of events that brought the record business to its knees by making music discovery instant – but payment optional. It was not the first service of its kind, but it was the one that went viral. Four months after its June 1999 launch, 150,000 people had signed up. By February 2001, it peaked at a verified 26.4m users, with some estimates topping 80 million.

It wasn’t just some student project gone rogue. From the earliest days, it was built as a business. Shawn watched his uncle John Fanning try to get rich in 1997 when he was hawking chess.net, an online chess service; John convinced his nephew to hand him 70% of Napster in order to help him make it a success.

The company quickly created a buzz among San Francisco tech investors during the dotcom boom. Venture capitalist Eileen Richardson says downloading the software was a Damascene moment: “What’s better than learning about new musical artists without having that big holy hell of a bunch of crooks called ‘the record industry’ in the middle?”

She and her friend Yosi Amram, who knew John Fanning from his chess.net days, put in money and met Fanning and Parker. “We definitely talked about [whether] this was legal or not,” she recalls, but Amram referenced the landmark Sony Betamax ruling from 1984 on home copying to try to allay fears. Richardson became the company’s first CEO.

Pioneer … Napster founder Shawn Fanning arrives at court in 2001. Photograph: Justin Sullivan/Getty Images

There is an ossified narrative that record labels were too arrogant and lazy to see the digital revolution coming, and deserved their downfall. But the story is also about the slowness of a generational handover of power. EMI’s Capitol imprint was the first major to sell a download version of a single, Electric Barbarella by Duran Duran, in 1997. But it was a Sisyphean struggle to get the boards to really listen. Jeremy Silver, VP for new media worldwide at EMI at the end of the 1990s, recounts chasing one of EMI’s most senior executives into the toilets to speak to him about digital plans: “The only time I could get him to stay still and listen to me was when he was having a piss!”

Alison Wenham was heading AIM, a newly formed trade body for independent labels, when Napster appeared. Her account of a meeting between industry heads at Midem, the industry’s annual trade conference, in January 2000, sums up this divide. “A man from one of the majors banged the table and said the answer was to ban the internet.”

As the new millennium began, in San Francisco, a power struggle within Napster was threatening to pull the company under. Richardson knew John Fanning’s 70% stake would cause problems. “I tried to keep him out of operations day-to-day as much as I could,” she says. She claims he wanted to work with one firm valuing the company at $115m, but which Richardson felt would damage Napster’s long-term future. (John Fanning did not respond to multiple requests from the Guardian to speak about his time at Napster.

Richardson was planning a per-track download model – “a dollar a song” – echoing the $0.99 track price that iTunes would offer in 2003. “Why would you spend $17 on a Britney Spears CD and get only two songs that you like?” she asks. “Wouldn’t you prefer to spend $17 and get 17 songs that you like?”

But at the time, breaking apart the CD was never going to land well with labels, and Richardson stepped back in May 2000. Music copyright lawyer-turned-VC Hank Barry became the company’s second CEO in a year. His company invested $15m and got two of the three seats on Napster’s board. “I just wanted it to survive,” says Barry. “I figured if the company could survive for a year or so and make some arrangements with the record companies, that it could be a very good investment.”

Napster had swiftly become a cultural talking point. By October 2000, its creator was on the cover of Time magazine. Moby said he was “veryflattered” that his music was being shared on Napster and Courtney Love claimed that “major label recording contracts” were the real pirates. Napster had also spawned copycat services such as LimeWire that were built with one eye on avoiding the same legal mistakes Napster had made.

But Napster was by now seriously under threat. The RIAA, the trade body representing US record companies, had begun suing the firm in 1999, and acts including Dr Dre and Metallica tried to close it down. In May 2000, the latter delivered 13 boxes of documents to Napster’s offices, listing hundreds of thousands of users suspected of unlawfully sharing their music.

Lars Ulrich of Metallica arrives at the California offices of Napster in May 2000 to protest against its users sharing his music for free. Photograph: Ben Margot/AP

But Barry was meeting with major labels, desperately hoping to hammer out some kind of deal. He got into advanced talks with Edgar Bronfman Jr, of Vivendi Universal, that June. “We had spreadsheets that covered a conference room table, trying to work out what the business model could be,” Barry says. He claims they were “48 hours away from a deal” – but then Bronfman stopped taking his calls. “To this day, I don’t know what happened.”

Silver suggests the real reason the majors did not want to do a deal with Napster was because doing so would blow a hole in their obscurantist accounting policies.

“Suddenly, the data associated with digital transactions was unambiguous [compared with traditional sales] and very powerful,” he claims. “For years, record companies had traded on dodgy numbers. What [the lawyers] were afraid of was wholesale renegotiation of artists’ contracts triggered by data transparency.”

While majors Universal, Sony, EMI and Warner viewed Napster as a threat, the independents and BMG were more accommodating. “Our perspective was pragmatic,” says Wenham. “We didn’t condone ruthless piracy, but this seemed to be much more of a fan-based phenomenon that had grown to tens of millions of users.”

Screenshot of the Napster interface. Photograph: pixel.pix/Alamy Stock Photo

Helen Smith was director of business affairs at AIM and, alongside Wenham, negotiated the deal on behalf of the indies. She wanted “the usual,” she says. “A good royalty rate, in advance, equity. Napster definitely wanted to do a deal. There were some who felt you shouldn’t do a deal with the devil, but they were the exception.”

Wenham says the business model – based on converting a percentage of Napster’s 80 million users into paying subscribers, similar to the freemium model Spotify established – was viable.

Napster had carried out audience research to determine how many users would pay for music and what they would pay: “If only 2% of the Napster user base converted to paying, it would be incredibly valuable to the sector.”

None of this could save Napster. Rolling legal action against it since December 1999 hobbled its growth and the resulting legal bills drained its resources. In September 2001, Napster agreed to pay copyright owners £20m in a settlement, and by May 2002 it couldn’t pay its staff, who were offered the choice of being laid off or taking a week’s unpaid holiday. Mass resignations ensued. The company filed for bankruptcy protection in June 2002 and the assets were eventually sold to Roxio. It relaunched as a legal service in November 2002, but by then, it was yesterday’s brand.

It took until 2015 for the record industry to finally move into recovery – watching aghast as its value dropped every year from 2000 to a low of $15bn in 2014.

The services slipstreaming in Napster’s wake benefited from the risks it took. Swedish technologist Daniel Ek was hugely inspired by Napster when developing Spotify, so much so that he brought Sean Parker on to his team. Spotify has just passed 100 million paying users (of 217 million users in total); the record business is now back to a £15.15bn ($19.1bn) value last year, though still only half of its 1999 peak.

Arguably, Napster’s true benefit was clearing out industry dead wood . “Some people just can’t get used to change, and that means we have to wait for those people to go away,” says Paul Hitchman, who set up digital music service PlayLouder as Napster was emerging. He is now president at label services company AWAL. “In a way, Napster won. From a consumer point of view, it opened up music to the whole world in a completely liberated way.”

The battle with Napster also instilled an image of the music business as grasping and arrogant. “Even today, the music sector are perceived as control freaks, living in the old world and trying to hold back innovation,” says Smith. “Obviously, that is not the case, but that image still haunts us today. That is one of the biggest downsides of what happened with Napster.”

Those who tried to make Napster a success still feel the music industry missed a great opportunity. “Shawn Fanning and I had a debrief a couple of years later, and we both decided that we were the first and that we were going to be made an example of – no matter what,” says Richardson.

Barry remains sanguine. “Shawn Fanning did an amazing thing, and was quite far ahead of his time,” he says. “But timing is everything, right?”