Which experiments can we trust the most?

In this episode of Datadrivet, Joni Lindgren and Jasmin Yaya lay out a hierarchy of evidence: a ranking of the ways teams try to find out whether an idea will actually work, from the weakest signal to the strongest.

They climb the ladder one rung at a time:

• Expert opinions. Leaning on expert views or industry best practice is not enough to prove an idea will work for you. Those opinions may not apply to your specific product or your users.
• User research. Interviews and focus groups give you real insight into the problems users face, but they involve too few participants to draw reliable conclusions about a larger group.
• Data and science. Open your web analytics tool and see how users behave in reality. Looking at the recent four weeks of data shows you traffic sources, device usage, conversion rates, and actual behavior rather than guesses.
• A/B testing. The hosts call this the crown jewel of evidence. Two versions, A and B, run at the same time under identical conditions, so you compare them in real time on real users.
• Meta-analysis. The top rung is A/B testing the A/B test: verifying a result by running further tests on the same hypothesis before you trust it fully.

The through-line is that confidence should match the strength of the evidence behind a decision. An expert hunch and a confirmed A/B test are not the same weight of proof, and knowing where your evidence sits on the ladder tells you how much to lean on it.

The takeaway: rank your evidence honestly, and climb toward real-world data and controlled tests before you commit.

Listen to the full episode of Datadrivet for the full walk through the hierarchy.