Only a few months ago Netflix Inc. Chief Executive Reed Hastings was on top of the corporate world. On Monday, he was forced to retreat from a strategy that backfired.

The Internet movie-rental company said it was abandoning a plan to split its DVD-by-mail service into a separate business named Qwikster, originally hatched so it could focus on its online video-streaming service. Customers had howled about that plan, which promised to heap inconvenience on top of a previously announced 60% price increase. Between the two moves,...