Whether we work in cubicles, the C-suite, or a home office, we’re always navigating the people and cultural norms shaping our workday.

In the modern workplace, autonomy is now king.

Research has shown that personal autonomy at work increases job satisfaction, which leads to higher engagement, and correlates with lower employee turnover. Given the choice, many employees prefer working remotely and having some control over their schedules. In some professions, like programming, more are working from home than ever before; some are even beginning to demand it. The idea has gained so much traction that even Japan’s highly traditional workplaces are adopting it.

But all that autonomy might come at a cost. A new working paper shows that when workers deviate from an organization’s prescribed schedule of when they should complete their tasks, there is a drop in productivity.

The study—by researchers from Harvard Business School, the University of Texas at San Antonio, and UNC Kenan-Flagler Business School—used data from over 2.7 million cases processed by a large firm that outsources radiological services. Doctors working individually at computer workstations and are randomly assigned images from X-rays, CT scans, MRIs, and ultrasounds to analyse from the company’s centralized queuing system. The researchers tracked when and how the images were received and processed.

While the company had an implicit policy that doctors should do tasks in the order they were assigned to them, the researchers found that doctors in the study strayed from their prescribed scheduling over 42% of the time. They tended to bunch together similar tasks into batches—analysing all chest X-rays together for instance— and also tended to avoid doing difficult cases first, preferring to get easy tasks out of the way. More experienced doctors were more likely to stray from the schedule.

They said that doctors making up their own schedules meant that the tasks took 13% more time per task than when they did them in the order that the were assigned. As a result, in a given amount of time, doctors were able to analyse fewer images.

This decline in productivity correlated with experience—the more experienced the doctors were, the less their productivity was impacted by using their discretion about the order in which to do tasks. However, the productivity decline still remained significant for even the most experienced workers.

While the study did not determine exactly why this was happening, researchers suggest that one possible reason could be the time taken to make the decision of what to do next, in order to improve the speed of work, actually hampers productivity. “Searching through your queue and deciding which task to choose next may not seem like it, but it’s actually taking you a long time,” said María R. Ibáñez, a doctoral candidate at Harvard Business School, and the lead author of the study. The study is currently a working paper and is due to appear in a forthcoming issue of Management Science.

The nature of the tasks in this study also varied quite a bit, from sorting to diagnosing. While taking the decision-making out of daily work might improve productivity for employees performing routine tasks, the evidence suggests that creative people’s brains work differently and the results might look different for companies whose employees perform more creative tasks.