As Sephora has emerged as the hottest retailer in luxury beauty, cosmetics companies increasingly are losing out—to Sephora.

The rapidly growing beauty chain has been on a tear, opening new stores and increasing sales, a welcome alternative venue for companies such as Estée Lauder Cos. and L’Oréal SA as department stores lose traffic and close locations.

But...