Which experiments can we trust the most?

In this episode of Datadrivet, Joni Lindgren and Jasmin Yaya rank the evidence teams use to find out whether an idea will work. The ranking runs from the weakest signal to the strongest. They call it a hierarchy of evidence.

• Expert opinions. Expert views and industry best practice do not prove an idea will work for you. They may not apply to your product or your users.
• User research. Interviews and focus groups show you the problems users face. They involve too few people to draw reliable conclusions about a larger group.
• Data and science. Open your web analytics tool and see how users behave. The recent four weeks of data show you traffic sources, device usage, and conversion rates.
• A/B testing. The hosts call this the crown jewel of evidence. Two versions, A and B, run at the same time under identical conditions. You compare them on real users.
• Meta-analysis. The top rung is A/B testing the A/B test. You run further tests on the same hypothesis before you trust the result.

Match your confidence in a decision to the strength of the evidence behind it. An expert hunch and a confirmed A/B test carry different weight.

Listen to the full episode of Datadrivet for the full walk through the hierarchy.