Worries about income inequality, jobs disappearing due to automation and environmental sustainability are all feeding wide-scale distrust in capitalism as the world knows it, according to a new study released Sunday.

Edelman, a public-relations firm, conducted its 20th annual analysis of public trust in major institutions, surveying 34,000 people in 27 countries and Hong Kong. The data reveal both skepticism about those institutions—including government, business, the media and nongovernmental organizations—and a hunger for...