How to get started with experiments

In this episode of Datadrivet, Joni Lindgren and Jasmin Yaya walk through how to get started with experiments. They cover the types of tests available and a weekly loop you can run.

They lay out three main experiment types: surveys and qualitative research, painted door or smoke tests, and A/B tests. A/B tests get used in three ways: pure optimization, checking that a release doesn’t introduce negative effects, and validating new ideas. Validating new ideas is the main reason to reach for them.

The hosts point to Microsoft research from 2009 on high-performing teams. Those teams had no better developers and no better ideas. They discarded the ineffective ideas faster. Across a product team’s work, roughly one third had a positive impact, one third had no effect, and one third had a negative impact.

Win rates at Microsoft, Pinterest and Slack sit around 30%. Facebook sits at 15%, and runs about 22,000 experiments at once. Google sits at 10%, and runs around 300,000 a year. The more innovative the test, the lower the win rate, but the higher the upside if it lands.

Collect all the ideas and turn them into hypotheses. Define minimum viable tests you can run in half a day to a day. Prioritize the hypotheses as a team. Then maximize your Build, Measure, Learn loops weekly: build minimally, test fast, and scale what works.

Listen to the full episode of Datadrivet for the breakdown of test types and win rates.