Flex-time is not as flexible as you might think.

The belief that getting an early start to the day is virtuous is widely held. In fact, finds a forthcoming study, it’s so pervasive that managers rate workers who get an early start higher than those who get in and stay late, no matter how many hours they work in total or how well they do their jobs. And it could explain why other research has found that workers who have flexible schedules have less successful careers.

The study, from researchers at The University of Washington, highlighted at the Harvard Business Review, will be published later this year in the Journal of Applied Psychology. It finds support for the idea that managers have a “morning bias.” In other words, they buy into a common stereotype that leads them to confuse starting time with conscientiousness. They perceive employees who start later as less conscientious, and consequently less hard-working and disciplined, and that carries through to performance ratings.

The researchers surveyed 229 employee/manager pairs—most but not all in the US—looking at when both the employee and the manager got to work, and how the manager rated the employee’s conscientiousness and performance. Start times for the employees ranged from 5am to 9:45am, with the average of 8:42. Even controlling for total time worked and the typical starting times in each workplace, people who started later were rated worse.

In a second experiment, undergraduates from a US university were asked to assume the role of a manager at a fictional company and rate an employee’s job performance. Participants were given identical profiles describing the employee’s performance based on contracts fulfilled, but the time the employee started work was varied. Late start times led to significantly lower ratings, even though productivity and total hours were exactly the same.

This finding is a particular worry for people who work flexible hours. Flex-time is often seen as a perk, letting employees structure their workdays around family or other commitments. But employees who start later, even for a good reason, might be inadvertently hurting their career prospects. And companies that put pressure on employees—even unwittingly—to start earlier are likely to lose a lot of the benefits of allowing flex-time in the first place, such as attracting talented people who might not otherwise be able to work full time, and letting people work when they’re most productive.

The study had one piece of good news for night owls, though. Managers who started later themselves were less likely to show “morning bias” when evaluating employees. So if you’re a late riser, try to work for a late-rising boss.