When British makeup artist Pat McGrath launched her own cosmetics line two years ago, few would have predicted that her self-financed company would so swiftly overtake established beauty brands. But this month Pat McGrath Labs became a billion-dollar beauty behemoth, valued at an astronomical 17 times retail sales – and it wasn’t the only one.

On the cover of Forbes in the same week, 20-year-old Kylie Jenner was audaciously declared a “self-made” billionaire, propelled by Kylie Cosmetics, which generated sales of $420m (£360m) in its first 18 months. Neither is available to buy on the high street, but both are among a handful of personality-driven “indie” brands – including Huda Kattan, Rihanna’s Fenty and Charlotte Tilbury – that have transformed the beauty industry in the past decade. So where did the likes of Estée Lauder, L’Oréal, Revlon et al go wrong?

“Innovation,” says Sarah Jindal, a senior global analyst at Mintel. “There is a continual race to be the first to do something and get it on the market as soon as possible, which is born out of smaller indie brands that have the flexibility to do new and interesting things.”

The frenetic pace of consumer trends and demand for newness have caught established houses, which have much longer lead times, off guard. Meanwhile, fast beauty brands such as Colourpop can get customer votes on a new lip colour on Monday and release the product by Friday. Social media – “especially Instagram”, says Jindal – has been the other major disruptor.

Ten years ago dedicated beauty junkies with knowledge of the best lip products or contouring tricks were considered a loyal minority of the cosmetics customer base – now they are the mainstream, democratising the market via Instagram, YouTube tutorials and beauty blogging. Mintel estimates there are 15.3 million beauty vloggers online, while market researcher NetBase says beauty brands were the subject of 5% of all social media conversations in 2017 (fashion brands made up just 2%).

Charlotte Tilbury, another makeup artist with a celebrity following. Photograph: Richard Young/Rex/Shutterstock

Huda Kattan, 34, named one of the 25 most powerful people on the internet by Time magazine last year, is perhaps the prime example of this new, but dominant, influence: a makeup artist and beauty blogger with 26 million Instagram followers, she amassed a hugely loyal audience in three years via her blog before launching Huda Beauty in 2013 and making her fortune – estimated at $550m – in false eyelashes.

Jo Osborne, beauty buying director at Harvey Nichols, believes another driver behind the success of the newer companies is the wider choice they have made available for women of colour – customers who have been underserved for decades by the industry. “Fenty launched with 40 foundation shades [double the range made available by older cosmetics companies] and has been the number one beauty launch in the history of Harvey Nichols.”

The department store has been one of the industry’s go-to “incubators” of beauty brands and was the first to launch Mac, Armani, Trish McEvoy and Shu Uemura in Europe. But the line launched by Rihanna last September “for the inclusion of all skin tones” has been a gamechanger: queues for the Fenty counter lasted for months and in the first few weeks snaked around the Harvey Nichols building in Knightsbridge.

McGrath, 48, has often spoken out about the sheer lack of makeup ranges available for non-white skin tones. Growing up, she said, “there was nothing” – she and her mother would have to endlessly mix makeup products to create the right shades.

Despite being considered the world’s most influential makeup artist and certainly the most prolific (McGrath creates up to 80 new catwalk looks each year and is on the road with a team of 25 to 90 junior artists for months at a time), she has spent much of her career frustrated by the attitude of beauty companies. “There is a whole planet out there,” she said. “How can you not address the whole world – what are you thinking?”

It’s partly this vacuum that has allowed newer startups to flourish. The global colour cosmetics market is estimated to reach sales of more than $47bn in 2018, marking year-on-year growth of $2bn. Celebrities dipping a toe into the beauty industry might once have landed an endorsement or fragrance deal, but now they are creating feverishly sought-after products: Jenner’s company-launching lipkits famously sold out within a minute of going on sale. The limited editions of McGrath’s early products in 2015 sold out on her own site, which encouraged her to create a fuller, equally hyped line.

Charlotte Tilbury, another makeup artist with a celebrity following – the likes of Kate Moss, Nicole Kidman, and Penélope Cruz – says the “nerdification” of beauty can only grow. “Before I launched [Charlotte Tilbury Beauty] in 2013, I discovered research conducted by Harvard University that revealed that 50% of women don’t engage with makeup at all,” she says.

“It was a big incentive and the motivation for me to start my brand. Ultimately I want to create my own limitless makeup revolution for all women.”