How experimentation is becoming the backbone of Plick’s growth
This case study is a detailed one. Thanks to the Plick team for sharing their journey and learnings so generously!
The CEO of marketplace app Plick felt an urgency to deep dive into what actually drives user engagement, monetization, and retention. Together with scilla.studio, they have:
developed relevant KPIs for the product teams to follow;
learned how to work with experiments;
implemented the process of testing everything they build.
“For me, working with product experiments is not optional. It’s a must! With every dollar we invest in acquiring a new user comes a high risk of losing that new user within the first few days. Until now, this has been a gaping black hole for us, but with scilla.studio’s help, we no longer waste time and money on meaningless guesswork.” (Jimmy Heibert, CEO)
After six months of collaboration and eight experiments per month, the product team learned that only a mere third (!) of what they built actually worked.
Rewind to fast-forward: The background story
Second-hand marketplace Plick has always been data-driven in its marketing operations. Together with a digital marketing agency, they use the tactics that one would expect from a start-up like Plick: monitoring, segmenting, targeting, the whole nine yards.
But when it comes to product development, CEO Jimmy Heibert felt an urgency of deep-diving into what they build and find out what moves the needle for key metrics. How could they improve their working methods to ensure that new UX and features benefit the users and the business?
Glad you asked, Jimmy! Luckily, these are the exact problems we help solve with some extra muscle. When Jimmy heard that scilla.studio was open for business, he wanted our help right away.
To kickstart 2021, scilla.studio joined the Plick product team with:
1 growth product analyst
1 growth product manager
Plick’s team consisted of a product manager, designer, developers, and engineers.
Experimentation step #1: How do we think our product will grow?
To get started, we worked on developing Plick’s growth model together. A growth model is a visual representation of how we think the product will grow and sustain its customer base, with assigned key metrics to measure success.
The growth model also helps predict what changes might have the highest impact on growth. Yup, it’s pretty cool but also a bit hypothetical.
[💡 You will soon be able to play around with a sample growth model here]
Experimentation step #2: Training is the mother of all skills
When the growth model was in place, and data collection was completed to feed the growth model, we were waiting to see the baseline.
In the meantime, scilla.studio started training the team in the experimentation process. You can't just add one “experiment person” to the team and think you’ll succeed. It’s a team effort!
The entire team has to shift their idea of how product development can be done and adjust to a slightly different working method. Together, the team must acquire and understand the experimentation knowledge — mindset and process. If not everyone is on the same page, you’ll fail to gain value from experimentation.
The team training took place during the next four weeks. This is hands-on, practical training where we combined scilla’s expertise and experience with the team's skills:
Data scientists master how to look at behavioral data and leading indicators.
Designers learn how to prioritize user problems over visual solutions.
Developers learn how to build as little as possible to validate hypotheses instead of writing scalable code.
Engineers investigate how to adapt the back-end and release pipelines to make testing possible.
And all of them, of course, learn how to think in hypotheses and prioritize testing based on potential impact instead of someone's gut feeling or story points. At the end of the training, they’ve shaped the experimentation process to fit the team.
Experimentation step #3: Learning how to lead the process
On top of the team training, Product Manager Sara Collin got individual coaching to ensure that her team could continue the work when scilla.studio was no longer on board. Sara does nothing by halves, and she always goes that extra mile. The coaching sped up her understanding to lead the product team in an experimentation process.
“It was a bit challenging at first! Everyone agreed on experimentation being something we should do, but we all had different ideas on making it work for us. We had to sit down, look at the recommended process and then make it ours,” Sara remembers the first weeks.
In just a few weeks, they:
set up tracking to collect relevant data;
built a dashboard to follow test results;
started their first experiments;
were ready to reflect on their learnings.
Scilla.studio always recommends holding a Growth Demo presentation every other week. Anyone in the company should be able to attend, learn what hypotheses have been tested, what the results were, and what’s next.
A Growth Demo is an excellent opportunity to share learnings and inspire more people in the organization to understand the data and insights. It’s also an essential forum for people outside the product team to pitch in with ideas on what might improve the metrics. These discussions help align on the metrics that matter because you have to understand metrics first to talk about results!
Result: 13% increase in revenue & learning how often you’re wrong
For one of the first experiments, the team set up a quick testing process for in-app promotions in just a few hours. By testing how to clarify paid add-on services in the app, Plick increased revenue by a whopping 13% in just two weeks.
However, we must underline that the experiment confirmed that some tested solutions did not increase revenue. We truly believed in one solution, but it failed so badly that it was worse than the original one. What if we just implemented that without testing? Just a few weeks back, we would’ve done it without blinking an eye. It would’ve taken the team months, if ever, to learn that the change caused the revenue loss.
It’s also important to mention that the win rate (the percentage of successful hypotheses, better than the original) of all experiments Plick has run since the start is about 30%. That means that only a third of everything they believed would work actually did work.
Do you want to read some more scary stats? A third of the tests has proved no impact, whereas a third negatively impacted the growth metrics. Can you believe that a third of everything you do actually hurts users and/or your bottom line? And can you believe that most product teams have no clue about this?
“I’ve learned that I can’t predict what will work. Things I believe to be great have proved no impact, and things I didn’t believe would make a difference were super important for the users.” (Sara Collin, Product Manager)
“One of the most tangible differences between working with validating hypotheses instead of shipping fully built features is how fast we can get new experiences in front of the users to learn what works. Build as little as possible, let the users experience it and measure the outcome.” (Jimmy Heibert, CEO)
What's next for Plick?
“We’re in a growth phase and will expand the organization significantly, so we’ll scale this way of product development as the organization grows. Coaching Sara was a great investment since we’ll be able to run this process on our own.” (Jimmy Heibert, CEO)
“Experimentation is now a fully integrated step in our product development process. The mindset of always validating what we believe, not only on the product team but the entire company.” (Sara Collin, Product Manager)
What’s your most valuable learning?
“I thought I was going to say that the most important impact would be the test results, but honestly, it’s the shift in mindset and how we make decisions. It's not ‘sexy’ talking about organizational changes and mindset, but I know for sure that every decision we make from now on will have a more significant effect on our key metrics, and this can always be traced back to us making this shift in how we work.” (Jimmy Heibert, CEO)
Who do you recommend to work with scilla.studio?
“Any company that believes that the product is the most important driver for company growth should get in touch with scilla.studio ASAP!” (Jimmy Heibert, CEO)