Netflix Inc.’s executives were torn. On the one hand they trusted the company’s algorithm. On the other they were worried about ticking off Jane Fonda.

After the streaming-video giant released the second season of the comedy “Grace and Frankie” in 2016, its product team put up an image to promote the show to U.S. subscribers that only included Ms. Fonda’s co-star, Lily Tomlin. Tests showed that more users clicked on the show when the photo didn’t include Ms. Fonda.

The...