Back in 1991, Warren Buffett met Bill Gates, though as he tells Levo League , neither of them were excited to see one another. But it turned out they had a great time talking–and during the course of the conversation, Buffett pulled out the little black date book that he carries in his pocket.

He flipped through it: The pages were practically empty.

“You’ve gotta keep control of your time,” Buffett says, “and you can’t unless you say no. You can’t let people set your agenda in life.”

Buffett, now 82, admits that it doesn’t get easier as you get older. But even if you’re not inclined to decline–like when your friend asks you to attend something–you need to develop the ability to say no. He’s far from alone: As Kevin Ashton wrote in a recent essay for Medium, being stingy with your time is part of leading a creative, productive life.

Ashton, who coined the phrase “Internet of Things,” observes a common thread between ur-manager Peter Drucker, novelist Charles Dickens, and photographer Richard Avedon: All of them guarded their time. Why? For in order to do your work, Ashton observed, you must have time:

“Wipe away the magic and myth of creating and all that remains is work … No matter what you read, no matter what they claim, nearly all creators spend nearly all their time on the work of creation. There are few overnight successes and many up-all-night successes.”

This is why if we want to do the work that we want to do, we need to own our time–how else can we spend it productively? As Reddit CEO Yishan Wong explained in an epic, lesson-filled Quora thread, time is limited in three ways: