Over the past decade, we researched the micro-level causes behind this macro-level problem. To gain real-time perspective into everyday work lives, we collected nearly 12,000 electronic diary entries from 238 professionals in seven different companies. Our study charted each person’s psychological state each day, and asked respondents to describe one event that stood out during that day. Our analysis revealed their inner work lives — the usually hidden perceptions, emotions and motivations that people experience as they react to and make sense of events in their workdays.

The results were sobering. In one-third of the 12,000 diary entries, the diarist was unhappy, unmotivated or both. In fact, workers often expressed frustration, disdain or disgust. Our research shows that inner work life has a profound impact on workers’ creativity, productivity, commitment and collegiality. Employees are far more likely to have new ideas on days when they feel happier. Conventional wisdom suggests that pressure enhances performance; our real-time data, however, shows that workers perform better when they are happily engaged in what they do.

Managers can help ensure that people are happily engaged at work. Doing so isn’t expensive. Workers’ well-being depends, in large part, on managers’ ability and willingness to facilitate workers’ accomplishments — by removing obstacles, providing help and acknowledging strong effort. A clear pattern emerged when we analyzed the 64,000 specific workday events reported in the diaries: of all the events that engage people at work, the single most important — by far — is simply making progress in meaningful work.

As long as workers experience their labor as meaningful, progress is often followed by joy and excitement about the work. “This time it looks good! I feel more positive about this project and my work than I’ve felt in a long time,” one programmer wrote after she’d completed a small but difficult task. This kind of rich inner work life improves performance, which further supports inner work life — a positive spiral.

Unfortunately, many companies now keep head count and resources to a minimum and this makes progress a struggle for employees. Most managers don’t understand the negative consequences of this struggle. When we asked 669 managers from companies around the world to rank five employee motivators in terms of importance, they ranked “supporting progress” dead last. Fully 95 percent of these managers failed to recognize that progress in meaningful work is the primary motivator, well ahead of traditional incentives like raises and bonuses.