How many experiments should we run?

This episode of Datadrivet answers a listener question: how many experiments should a team actually run? Joni Lindgren and Jasmin Yaya give the short version up front, that you want as many as you can manage, because more tests mean faster insight, and then they unpack what that takes.

They sort teams into rough maturity stages. A team in its infant phase runs somewhere between 1 and 5 experiments a month. A more advanced team can reach up to 1,000 experiments a year. The gap between those numbers is the gap between guessing occasionally and learning continuously.

The reason volume matters comes down to a hit rate. Only about 10 to 20 percent of ideas actually succeed, so if you only run a few tests, you may go a long stretch without finding a winner. Running more raises your odds of landing on the ideas that work.

Volume is not free, though, and the hosts spell out the team you need to sustain it:

• An analyst to set up and analyze the tests
• Developers to build the experiments
• UX designers to design them
• A team leader
• Cross-functional expertise that can move quickly together

They also discuss a question that comes up once an experiment wins: who should build the finished version, the experimentation team itself or another department that owns the area.

The takeaway: aim for as much experiment volume as your team can sustain, staff it properly, and let the hit rate work in your favor.

Listen to the full episode of Datadrivet for the full discussion on pace and team setup.