It was a fluke that the Netflix co-founder Marc Randolph changed the history of television. It almost didn’t happen.

In 1997, the Santa Cruz businessman was spending his carpool rides to work brainstorming internet startup ideas with a colleague. They discussed personalised surfboards, customised dog food, shampoo by mail. One commute, the chat turned to “videotapes”.

Randolph’s three-year-old daughter had struggled to sleep the night before, leading them to watch a used copy of Aladdin. His car companion was intrigued, having recently received a $40 Blockbuster late fee for Apollo 13. Could they make it easier to rent movies?

Randolph soon after launched Netflix, an initially unsuccessful movie-rentals-by-mail service that went on to upend Hollywood and draw more than 150 million subscribers. In his new book, That Will Never Work, the 61-year-old offers a glimpse into the tumultuous early days of Netflix, which began as an obscure Silicon Valley startup, resisted pressure to sell to its online retail competitor Amazon, defeated Blockbuster and eventually evolved into cultural force that fundamentally changed the way we consume and create media.

Marc B Randolph co-founded Netflix, which launched in 1998 with about 800 titles. Photograph: Dan Tuffs/The Guardian

Seated at a noisy coffee shop in Los Angeles, Randolph says he never dreamed of disrupting the entertainment industry. He now thinks his various 1997 carpool ideas were “all equally good and equally bad”. At the time, Amazon was demonstrating that seemingly absurd ideas were possible: “You could take a bookstore – a bookstore! – and make that work online.”

Randolph had been involved in software companies and developed the wealth and connections he needed to pursue new ideas. A merger led him to Reed Hastings, the entrepreneur who became his co-founder and is now Netflix’s CEO.

The two landed on movies after their carpool talks, but nearly abandoned the idea when they calculated VHS shipping costs. They soon, however, discovered DVDs, then a nascent technology. They secured investors, including $25,000 from Randolph’s mother (“My mom was delighted that I was finally doing a company that she actually understood”), and Netflix launched in April 1998, with about 800 titles in its inventory and Randolph as CEO.

The name Netflix, which Randolph thought was somewhat “porn-y”, beat E-Flix.com, CinemaCenter, NowShowing and others, which now sound like particular relics of the 90s.

Randolph recently found notes from a speech he gave when the company started, saying, “‘In three years, we want to be one of the top 10 video chains.’ How lame is that? I wanted to be as big as a single Blockbuster store.”

In 1998, Jeff Bezos attempted to acquire Netflix as a way to jumpstart Amazon’s entry into video. Even though Netflix was making a majority of its revenue from DVD sales, not rentals, Randolph and Hastings decided not to sell to Amazon or try to compete with it, and instead focus solely on rentals. It was a strategy that paid off. Eventually.

Netflix grew its subscriber base by offering free trials and other deals that made the convenient rental service extremely popular, but meant it was also haemorrhaging money.

Blockbuster finally agreed to talk to Netflix, calling an unexpected meeting the morning after an alcohol-fuelled Netflix retreat. Randolph says he was wearing shorts, a tie-dyed T-shirt and flip-flops when he and his colleagues sat down with Blockbuster in Dallas and proposed the video chain accelerate its entry into DVDs, by purchasing Netflix – for $50m.

Marc Randolph is a fan of Narcos, one of Netflix’s hit series. Photograph: Juan Pablo Gutierrez/Netflix

“In one fell swoop, we might get out of this,” he recalls.

After they stated the dollar amount, Randolph noticed something strange happening with the Blockbuster CEO John Antioco’s face. He was struggling not to burst into laughter.

The meeting went further downhill from there.

Randolph says it was one of the lowest moments for the company: “You fly to Blockbuster, try and sell the business, and they laugh at you.”

The only option left was to defeat Blockbuster, and Netflix stayed afloat by doing painful layoffs, figuring out overnight delivery of DVDs, and preparing early to move into streaming.

Hollywood executives were “very, very scared” of losing control of the content after the music industry had suffered at the hands of Napster, he says. “You have a studio who is doing $8bn in box office. Are they going to compromise [that] for half a million dollars in potential streaming? They had this whole nice system … that was working great. Why mess this up?”

He continues: “Because we can.”

Randolph left Netflix in 2002, which is also where his book ends, meaning tech elites and startup workers will have more interest in reading it than Orange is the New Black fans. Randolph also lacks some self-awareness in his discussion of Netflix’s early culture, fondly remembering a drunken raucous retreat, a new-employee tradition akin to “hazing”, and his insistence on maintaining a poster in his office that “wasn’t the most HR-friendly”.

Netflix resisted pressure to sell to its online retail competitor Amazon, and eventually defeated Blockbuster. Photograph: Geoff Moore/REX

Asked about the lack of diversity in Silicon Valley and whether he thought Netflix in the early days did a good job, he says he takes pride in the “gender”, “age” and “geographic” diversity of his employees, adding: “Diversity is not a skin thing, necessarily. Diversity is you have people around the table who have different backgrounds and different experiences and think differently.”

Does he think Netflix disruption will continue in other arenas? “Who knows what form storytelling will take in the future? … They begin doing virtual reality, holographic avatar games?” Some form of new disruption is certain, he says, though emphasizes that he is only speaking as a fan of Netflix (he like Ozark and Narcos).

Randolph argues that in the startup world, “no one knows what’s a good idea or a bad idea until you try it”. It’s a principle that is arguably relevant to Netflix today, which some critics say has a quality control problem and is creating an industry with an oversaturation of original streaming content.

As a viewer, he’s not bothered: “Why would I mind that there’s too much?”

And how does he reflect now on the demise of Blockbuster?

“I absolutely feel for people whose businesses have been disrupted, people who lost their jobs. That hurts.”

But, he says, “I’m not sure I want to preserve the old ways just for the sake of saying, ‘I don’t believe in change.’”

He tries not to gloat, but adds: “Blockbuster had 9,000 stores. Now there is one.”

• That Will Never Work – The Birth of Netflix and the Amazing Life of an Idea, by Marc Randolph is published by Endeavour (£20)