What data can tell you:

In this blog post, Airbnb Data Scientist Jan Overgoor describes how his team uses controlled experiments to inform every step of the product development cycle. He also addresses how experiments must be designed and implemented in a way that protects against various sources of bias.

Where could you add controlled experiments in your product development? How could you protect your experiments against bias?

What data can’t tell you:

In this article, Facebook Product Design Director Julie Zhuo describes some pitfalls of A/B testing that product designers are particularly vulnerable to and cautions against relying too heavily on experimentation to replace innovation, imagination, or decisive thinking in the face of uncertainty.

Which pitfall of A/B testing could apply to your current product development?

What users can tell you:

In this article, Google Ventures Design Partner Braden Kowitz argues that understanding and listening to your users is what fuels the design process and addresses the most common reasons that designers (still) don’t do it.

Which of the excuses listed have you or your team made to avoid talking with customers?

And in this post, Matt Gallivan, UX Research Manager at Facebook, maintains that there’s still no substitute for good qualitative research, even in the age of big data.

Think about what you might learn from observing how users interact with your product.

What users can’t tell you

Even when they want to, there are some things your users cannot tell you. Psychologist and Economics Nobel Laureate Daniel Kahneman spent his career understanding the cognitive biases that prevent users from accurately reporting their own experiences or motivations. In this review of Kahneman’s 2011 book, Thinking, Fast and Slow, New York Times reporter Jim Holt reviews some of these biases, which frequently affect qualitative product research.

One of Kahneman’s findings that I’ve found to be especially helpful is the distinction between the “experiencing self” and the “remembering self.” Designers and user researchers often prioritize the experiencing self. But as Holt learns, “what makes the ‘experiencing self’ happy is not the same as what makes the ‘remembering self’ happy” and, more importantly, “it is the remembering self that calls the shots, not the experiencing self.”

How could you modify your user experience to entice the “remembering self?”