SHANGHAI—Fifteen years ago in California, a tall technology geek named Steve Mushero started writing a book that predicted the American dream might soon “be found only in China.” Before long, Mr. Mushero moved himself to Shanghai and launched a firm that Amazon.com Inc. and Alibaba Group Holding Ltd. certified as a partner to serve the world’s biggest internet market.

These days, the tech pioneer has hit a wall. He’s heading back to Silicon Valley where he sees deeper demand for his know-how in cloud computing. “The future’s not here,” said the 52-year-old.

For years, American entrepreneurs saw a place in which they would start tech businesses, build restaurant chains and manage factories, making potentially vast sums in an exciting, newly dynamic economy. Many mastered Mandarin, hired and trained thousands in China, bought houses, met their spouses and raised bilingual children.

Now disillusion has set in, fed by soaring costs, creeping taxation, tightening political control and capricious regulation that makes it ever tougher to maneuver the market and fend off new domestic competitors. All these signal to expat business owners their best days were in the past.

The Trump administration is making a hard-nosed challenge to China using trade tariffs, investment controls and prosecution of technology thieves, and many in American business are cheering, if silently, having soured on the market after years of trying.