This is an adapted excerpt from Hiding in the Bathroom: An Introvert’s Roadmap for Getting Out There (When You’d Rather Stay Home), which is out now from HarperCollins.

It’s 2:30 p.m., and you’re stuck at your desk. The Slack channel is throbbing, the fluorescents are giving you a migraine, and you keep getting interrupted. You could probably finish your work if you could do it at home, but you’re up for a promotion. You want your boss to see you’re working hard.

Then you see your co-worker leave to pick up her kids, and then will sign on from home later. Lucky her, you think. But here’s the secret: Unlike you, she’d do anything to stay at the office, because she likes to be in the thick of it, gets distracted working from home, and is also aiming for a promotion.

The notion of working flexibly has gotten seriously mommy-tracked. Work-life balance and flexible work arrangements are so strongly correlated with parenthood that we assume a successful childless person prioritizes career advancement over anything else, and that a colleague who leaves early doesn’t take her job seriously. Both the parent furtively checking emails at soccer and colleague without caregiving responsibilities who’s stuck at a desk are in a system that equates face time with ambition.

When I quit my last full-time job, it was years before I got pregnant. I just couldn’t stand the hyperstimulation of the office environment. (And this was before Slack and Twitter!) I loved my work. I just hated going to the office.

Here’s the truth: Some working parents want to work all the time; some professionals without children want a less draining schedule​. We learned many years ago that, in the workplace, autonomy and control help engage people, whether you’re married, single, childless, or a parent. Accounting giant EY has now measured retention from a flexibility perspective for a decade, and they know that the top 25 percent of teams with the highest levels of flexibility—meaning they can determine where, when, and how the work gets done—have five points better retention than the bottom 25 percent.

Still, most women only begin to take advantage of official workplace flexibility policies when they have a caregiving crisis, whether for a baby, partner, or elderly relative, and it costs them. It intrinsically positions their flexibility as a perk. It strips power from the worker who uses it. It mommy-tracks us, when instead, our autonomy should be valued and productivity should be rewarded more than face time.

Male higher-ups, on the other hand, tend to use the system to emphasize their power. A recent study of 726 MBA men and women showed that, though both sexes preferred flexible hours, the men did not take advantage of formal telecommuting programs as frequently as the women. Instead, when they wanted to work from home or remotely, they simply did.

In this environment, it’s not surprising that women often start businesses to seize control of their time and schedule at work. But despite how great running your own shop can be, a Kauffman Foundation study shows female small-business owners pay themselves an average of $17,000 less per year than men who own small businesses, and the average revenues of a woman-owned firm are only 27 percent of that of businesses men own. When you add in that women’s careers suffer disproportionately for using flexibility and dialing back for caregiving (even earning hundreds of thousands less over their career), workplace policy clearly becomes an economic issue, and the price many people pay for taking advantage of reduced or flexible schedules seems ridiculously high.

But there’s hope. When I quit, I didn’t know a manager would do almost anything to avoid losing a valuable employee. Workplace expert Cali Yost has interviewed hundreds of managers, and she says the problem is they can’t understand when a worker simply says, “I can’t travel anymore,” or “I want to work from home.” What the employee needs to present is a plan: “Here’s how it’s going to help me, here’s how I’m going to do it, and here’s how people can reach me if you need me.”

Failing to negotiate a better schedule with a boss before leaving is like getting divorced without telling your partner you’re unhappy. But our work culture has made people afraid to ask for exactly what they need.

The ubiquitous cellphone (as we all know) can easily erode boundaries between boss and employee. For every employee checking emails at the playground, there’s one longing to throw the cellphone into the sandbox. Still, things that would have been considered shocking just 20 years ago (replying to your boss at 11:30 at night in a nonemergency situation, for example) are now quite normal. And here’s what’s really interesting: Americans don’t seem to mind feeling tethered to office work by their devices, as long as they don’t have to physically be at the office. Gallup polling finds that among U.S. workers who report they frequently check email away from work, 86 percent say it is a somewhat or strongly positive development to be able to do so. The same poll finds that when “asked how much time in a typical seven-day week they spend working remotely using a computer or other electronic device, such as a smartphone or tablet, employees who report checking their email frequently say they spend nearly 10 hours working remotely.”

There is no one size fits all, but most of us just want more control over our time and our workday. If you’re a powerful entrepreneur, that’s easy. But what about the rest of us? Who said it’s OK to ask to work from home without expecting a ding on career advancement? Most of us, myself included, can’t ignore pushback from our managers, clients, and colleagues, no matter how subtle it may be.

There’s a simple answer, but it’s not easy: Work needs to catch up with us.

In the 21st century, human capital will be the most valuable resource in our economy. As we recognize neurological and emotional diversity in all its forms, workplace culture needs to begin to make room for the “technicolor range of emotion,” as executive and tech founder Christina Wallace calls it. Although so much has been done (rightly) to promote diversity at work, there’s a giant hole in the understanding of how temperament and emotions play into not just our daily grind at the office, but the very trajectory of success.

It’s my fondest wish that managers and HR professionals also begin to recognize the ambivalence and inner conflict many insanely talented people feel. Because when they get the space they need, great employees have no reason to quit, or feel miserable.

I believe that if you do a great job and work hard, you’ve earned the right to control your own time and location at work. A promotion shouldn’t depend on sitting in an open-plan office 50 hours a week. But how many bosses agree remains to be seen.