Experiment, Measure, Repeat

Improving your company with continuous experimentation

A few months ago, I read an article from Michael Simmons where he essentially said that

“Deliberate experimentation is more important than deliberate practice in a rapidly changing world.”

Even though the article focused more on experimentation for personal improvement, the first thing that came to my mind was something like: “this should be one of the core values of every company” (here’s a Forbes article that tells the same).

Here at buildo, we take experimentation and continuous improvement seriously: it’s our core value. The goal of this post is to share with you how we do experiments internally, also adding some advice that you could follow, hoping to be useful for some of you that do experimentation or that want to start (you should!).

Let’s start with a simple case study

In October of the last year, we started noticing that some of our Slack channels were noisy and difficult to follow, especially in high work-load weeks. People were speaking about different topics in the same channel (e.g., two issues of the same software project), with unrelated messages alternating between one topic and another.

We decided to solve such pain point like we usually do: with an internal experiment. Luca Cioria, who proposed the experiment, became its owner and did the following:

defined a goal: “Make Slack easier and faster to read, considering the growing amount of messages.” defined how to try to reach the goal: “Use threads as much as possible, and create a culture of respecting channel’s quiet.” defined a metric to be measured to evaluate the success of the experiment: “The percentage of messages in threads.” defined success conditions for the experiment: “People like threads and accept to use them” and “there is a significant increase in the metric (set to 70% mostly as a psychological goal)” defined a date for the final evaluation (i.e., success or failure?): “2018, 10th week.”

Luca then presented all this to the rest of the team, which agreed to start; he then created a plot of the success metric and added it to our internal dashboard, to create awareness about the progress.

This is the result so far: