How to conduct experiments

There are several steps involved in the process of designing, prioritising, conducting and implementing experiments and the results of experimentation.

Ask questions and generate ideas Create a testable hypothesis Conduct your experiment Communicate your results Prioritise your results

1. Ask questions and generate ideas

All experiments start with a question. Your experiment is an attempt to answer your question through a series of hypotheses.

The question you ask should be linked to your product goals. For example, if your goal is to achieve a 10% visit to lead conversion rate on your sign up page, you could ask the question ‘what can we do to change the sign up form to increase the conversion rate?’.

If a product goal in an ecommerce product is to generate 15% more revenue year on year, your question could be ‘what is the best way to increase the average order value per customer from $20 to $30?’.

Questions are a form of ideation and which help to create hypotheses which can be tested.

The questions and ideas that you’re interested in exploring should ideally come not only from your own head but also from your stakeholders from across the business.

As a product manager, you’ll no doubt be in constant communication with many stakeholders across the business who (hopefully) know what’s on the roadmap and why.

How to involve stakeholders

If you do decide to conduct experiments, make sure you involve stakeholders at this stage and get them involved in the process of generating hypotheses by asking questions that are relevant to both them and to your product goals.

A simple way to do this is to create an experiment backlog which can either be a spreadsheet accessible by all or a Trello board where stakeholders can dump their ideas quickly and efficiently.

Decide on a structure for your experiment backlog and communicate the structure clearly to all your stakeholders. Make it clear that the structure must be adhered to for the experiment to be prioritised / understood effectively.

Stakeholders are busy too so ensure the structure you decide upon is easily understood by everyone. If you make the process of suggesting an experiment difficult you’re already setting yourself up for a fail. Keep it simple.

With a backlog prepared, you’ll need to prioritise.

Once your stakeholders have been involved in the idea generation and question generation process, you will then need to prioritise which questions / ideas to take forward.

You may have your own methods for prioritising experiments and there is no 1 definitive answer on the best methods for prioritising experiments for your product.

Here are some methods to help you prioritise your experiment ideas.

Impact vs effort – if you find out the answer to this question or you discover that this idea is one that should be implemented, what is the impact of this? What material change will happen as a result of this? Impact for your product may be different to impact for another. Be specific and attempt to articulate the best and worst case outcomes. In addition to the impact, consider the technical challenges or effort involved in implementing the experiment or indeed the final changes if the experiment itself can be done in light touch ways.

– if you find out the answer to this question or you discover that this idea is one that should be implemented, what is the impact of this? What material change will happen as a result of this? Impact for your product may be different to impact for another. Be specific and attempt to articulate the best and worst case outcomes. In addition to the impact, consider the technical challenges or effort involved in implementing the experiment or indeed the final changes if the experiment itself can be done in light touch ways. OKRs – how do these experiment ideas / questions link to your company or product objectives and key results (OKRs). If you don’t use the OKR framework, how do these experiment ideas or questions help you to achieve your short term and long term goals? Do they help you to achieve your goals? What will happen as a result of discovering the answer to these questions and how would your roadmap change? If the potential experiments don’t help you achieve your goals, why would you prioritise them?

You’ll no doubt use similar prioritisation frameworks for prioritising your overall backlog, so use the same principles here. As I said, much of this is guesswork since that’s how scientific experimentation works, but try your best to put together a prioritisation process that works for you.

Involve stakeholders at this stage if you think it’s appropriate. Work with stakeholders to determine which experiments to tackle. If needs be, get everyone together on a call or in a meeting so that you can discuss which of the ideas will be prioritised first.

Whilst stakeholders should have a say in what goes into the backlog, the product manager should make the final decision on prioritisation. Stakeholders will be more interested in the prioritisation of your roadmap and less interested in your experimentation backlog so if you think involving them in this stage might just confuse things, just crack on yourself and ping around a note instead to explain which experiments will be prioritised and why.