SHANGHAI—A downturn in China’s car market has wrong-footed some of the world’s biggest auto makers, saddling them with factories they no longer need and that are costly to retool.

Ford Motor Co. , Peugeot SA and Hyundai Motor Co. especially mistimed recent expansions, opening new plants just as the seemingly unstoppable growth of China’s auto market went into reverse.

It has gotten so bad that at one Peugeot factory, skilled workers spend their days washing floors or attending Communist Party political study sessions at work. At a Ford plant, workers’ shifts have been reduced to a few days a month, according to employees.

Now these auto makers face a painful dilemma: Abandon those big investments, or invest even more to turn around dying plants at an uncertain time in a crucial market.

“Looking back, it wasn’t the right choice” to build new factories, said Paul Gong, an auto analyst at UBS Group AG . “No one was willing to predict that they might ever lose market share in China.”