How to get started with experiments

In this episode of Datadrivet, Joni Lindgren and Jasmin Yaya walk through how to actually get started with experiments, from the types of tests available to a weekly loop you can run.

They lay out three main experiment types: surveys and qualitative research, painted door or smoke tests, and A/B tests. A/B tests get used in three ways: pure optimization, checking that a release doesn’t introduce negative effects, and validating new ideas, which is the main reason to reach for them.

The numbers in the episode make the case for running more experiments. The hosts point to Microsoft research from 2009 showing that high-performing teams didn’t have better developers or better ideas. They discarded the ineffective ideas faster. Across a product team’s work, roughly one third had a positive impact, one third had no effect, and one third had a negative impact. Win rates from the companies running these programs are sobering: around 30% at Microsoft, Pinterest and Slack; 15% at Facebook, which runs about 22,000 experiments at once; and 10% at Google, which runs around 300,000 a year. The more innovative the test, the lower the win rate, but the higher the upside if it lands.

The starting playbook is a tight loop. Collect all the ideas, turn them into hypotheses, and define minimum viable tests you can run in half a day to a day. Prioritize the hypotheses as a team. Then maximize your Build, Measure, Learn loops weekly: build minimally, test fast, and scale what works.

The takeaway is that experimentation is a volume game with discipline. Test more, kill losers quickly, and let the rhythm do the work.

Listen to the full episode of Datadrivet for the full breakdown of test types and win rates.