We all waste a lot of time. It’s hard not to — we work too hard, we’re tired, and sometimes our minds wander and we get distracted.

But, according to powerful figures like Bill Gates and Warren Buffett, the secret to success is to channel our energy toward the most productive past times and hobbies and stopping activities “with low returns” – the more time we waste, the less time we have for what really matters.

As Warren Buffett famously said, “It’s the only thing you can’t buy. I mean, I can buy anything I want, basically, but I can’t buy time.”

A 2009 study from the University of Adelaide’s School of Psychology found that people are “far less willing to keep many options open” when some have bigger rewards than others. And that’s an important message. Look at Bill Gates and Warren Buffett – they have learned to give up some activities to focus on the ones that are more rewarding.

“I stopped listening to music and watching TV in my 20s. It sounds extreme, but I did it because I thought they would just distract me from thinking about software,” the billionaire wrote in a blog post.

Buffett does not like distractions either, in fact, he doesn’t even have a computer in his office! By using a flip phone, he limits internet distractions. We don’t’ quite want to go THERE though…

Ultimately it’s not about being all work and no play, being productive can be fun, it just shouldn’t waste time or distract from your goals. For example Buffett likes to play bridge. “It’s got to be the best intellectual exercise out there,” he claims. “You’re seeing new situations every ten minutes. Bridge is about weighing gain-loss ratios. You’re doing calculations all the time.”

In the end, it’s all about moderation and knowing that the more time you spend on what doesn’t matter, the less time you will have for what does. Plan accordingly.