A respectful criticism.

If you’re a hardcore evangelist of “Lean UX” it’s possible that we’re about to have an uncomfortable moment.

The video on the other side of this link is a quick (2 min) video presentation of some general points about testing UX / Design stuff in your company.

My intent is not to criticize the whole approach in general. The description of Lean UX (linked above) is definitely a huge step forward from old-school methods, and is fairly consistent with my methods as well.

In Jeff Gothelf’s words, the most critical mistake that teams make is assuming they are on the same page as their colleagues, and the solution is to assess the risk of those assumptions.

To be fair, Jeff is promoting a mentality where you do a very minimal approach to everything, with lots of feedback, quickly and iteratively, in order to use minimal resources for maximum effect.

That’s fine. And smart, in principle.

But the way he presents this video promotes or implies three things that are major problems in UX, in my opinion:

1) Cost or “risk” as the primary factor in a design decision.

2) UX decisions happening at a team or company level based on internal feedback as a filter.

3) “Ideas” from the team being treated as “hypotheses”.

Let me break those down for you, because I think they are really important to understand fully.

Problem #1: Focusing on the cost of implementing instead of the cost of not implementing.

I understand it in theory, but I think Jeff is wrong in practice. It is well-established that people value the things we can have sooner more than later. Even when it’s irrational.

i.e. — We like doing things quickly because it’s quick, not because it is better.

The unfortunate truth is that sometimes the make-or-break things you should do are things that cost a lot of time and money. That’s scary, I know, but it’s true. I have worked in many companies — even start-ups — where those big, valuable things never get done, because 10 small projects feel like doing more than one big project.

Cost and risk are not the same thing in a good UX process!

The result of that is that you only make incremental changes in your product, because you’re not “allowed” to make game-changing improvements. It ends up being a pile of little ideas that add up to nothing and create a rats nest of confusion for users.

Problem #2: Letting internal feedback from non-UX people decide which UX choices are worth pursuing.



I get the sentiment, I do: these hypotheses are often strategic, so you need to get the whole team or even the whole company to believe in them, otherwise you’re spinning your wheels. I have soooo been there. But this video implies that your whole team must see eye-to-eye and the whole company must understand and agree before moving forward.

But there is a big difference between expertise and consensus. I strongly disagree that internal feedback is a step before external feedback from users, or that it is important at all before you have some solid reasons for making a suggestion.

The Lean process Jeff suggests puts a persuasion filter on everything. In other words, only ideas that are persuasive internally get tested externally, which is a recipe for excluding certain types of ideas.

Problem #3: Confusing possibilities with solutions.

Here we might be looking at the most subtle and most dangerous implication from the video. Everything about this sounds like the team has come together first and made a list of things they want to try. And only THEN do they go out and do something to determine whether their ideas are good or not.

First of all, that’s not a hypothesis. That’s creative whim. Only the most superficial features can be done like that. A hypothesis is based on research: observations, data, science, etc.

Second, that’s a process built to confirm yourself, not to reveal hidden information. You can’t test an idea; you can only test an implementation of an idea. If you could test an idea before it’s implemented, you wouldn’t have to measure the real thing when you launch.

If your research is based on pre-existing ideas, you might be leading yourself away from the best idea that you haven’t thought of yet!

Don’t wait until you have an agenda in mind to talk to people or get some data. That just adds bias. Use your interviews and data to form your hypotheses, and then choose the ones you want to test based on potential impact — from the customer’s perspective — and the strength of the evidence.

Ironically, if you follow Jeff’s advice you might end up testing the maximum number of low-quality options, in the lowest quality way. The Lean method is an awesome way of doing research, but a crappy way of testing actual solutions.

If you follow what I am suggesting (research first) you will test very few things — sometimes quickly and iteratively, sometimes not — and you will get very useful results almost every time.

So to summarize:

I think Jeff is promoting something solid from a macro level. A “lean” mentality is healthy for UX and business alike. But, like any orthodox approach to anything, it comes with (and possibly creates) problems you should be aware of.

1) Prioritize potential new features based on what they will add to your product and your bottom line. Not just the cost to build. “Hail Mary” ideas should not survive a proper UX process. But long term, expensive, time-consuming solutions, that are highly-valuable and well-supported by research, should.

2) It’s great to have an informed team and executives. Definitely. No question. But not so they can do the UX work themselves! Inform them so they understand and support UX in general. Consensus and expertise are very different things.

3) “A hypothesis is a proposed explanation for a phenomenon.” If you’re coming up with ideas before you have a reason for them, you might be wasting everyone’s time and money. Use the lean methodology to reveal hypotheses, not to test them.

You can buy Jeff’s book here. And you should watch the video.

