Defining experiments
Most of our products are always innovating and building things customers are asking for. This doc isn’t talking about those kinds of innovations we do every day.
For the purposes of this document, an experiment is a new product that is so different from what we do now it doesn’t naturally fit in one of our existing buckets. It’s a little bit of if it looks, smells and quacks like a duck, it’s probably a duck.
I’ve included little accents by Justina, who has led multiple experiments that have succeeded and failed.
How we choose
We look for experiments that:
- have a big market – needs to be able to impact our metrics, we want billion dollar opportunities if brand new or appeal to a wide spectrum of our customers. If a new product/business line, show me how this can be a 100m valuation company. If an add-on for customers, show me how this will benefit >30% of our customer base. If this seems hard, that’s on purpose – we have limited time and should maximise ROI on our time
- drive towards our goals/metrics, e.g. helps drive TPV by increasing engagement, increase product suite to deepen hooks
Experiment construct
Goal: must achieve 7% wow growth or 30% mom growth (+ bespoke metrics depending on line of business, e.g. NPL if lending). In our case this typically looks like getting a few customers to buy before we build and then finding more and more to continue to justify investment into the idea.
Team:
- 2-5 ppl, 1 technical, 1 product/sales minimum
- Must be willing to work 7 days a week until project succeeds or till I cancel (typically 3 months). I’ve found that within our org it takes this amount of sheer willpower and hustle to make something succeed from nothing. This also self-selects to see if you actually believe in the idea enough to devote many hours a week to it.
- These guidelines are roughly based on YC’s flagship program
“Each member of the team has to be willing to learn new things and adapt to make the project succeed, even if those things aren’t within their current skill set. Also, you need to be humble enough to take on any and all tasks required of you, even the really small and menial ones (and there’ll be a lot of these given how small the team is)”
Justina – Xenditer
Time constrained to 3 months – longer if Moses approves (typically where regulation is a precursor). After 3 months, we look at the project and see if you’ve hit targets or why you haven’t. If you hit targets, we assign more resources ($, devs, ppl) to it. If not, we axe the project gracefully, e.g. pivot it into something else or let it slowly die. We often maintain experiment projects that have sold even after the project so make sure you’re so committed to an idea, even if it dies, you’re happy to keep serving the <5 customers who are still using it.
Ideas can come from anywhere in the org.
Useful tidbits
You should start by speaking to real potential customers (at Berkeley we’re taught 30 and I think at least 10 in my experience) and find someone that will buy, i.e. hand on heart once the product is released they’ll use it. This is the cheapest/easiest/best time to iterate, when your idea is still an idea. You will likely face a lot of resistance from Moses if you have no real customers ready to use. At this point, I’d expect a few pivots of the original idea and lots of adjustments of the product. If nothing changes, you’re probably wrong (fooling ourselves or we’re not understanding our customer) Remember, startups are simple – build something people want.
Use (at least) the CCCP framework as you think about market entry. This means when you write up the 6 pager or present your findings this framework helps you organise your thoughts and make sure you cover the important topics.
- Customer – who are we trying to serve?
- Competition – what are the other options? What’s wrong with them?
- Company – why are we well placed to win?
- Product – so what should we be building? how do we build it?
- Success metrics – agree what success/failure looks like.
“This helps to not only keep the team on track but also take the sting out of the disappointment if the project dies”
Justina – Xenditer
The construct of the experiment is resource constrained on purpose. Many anecdotes around how restrictions drive creativity because you can’t rely on the usual methods. One of my favourite examples is how 800 Aussie light horsemen defeated a whole Turkish army when the British army couldn’t at Bathsheba in WW1. With resource constraints, comes a lot of questions around capacity and where to prioritise time. One rule that I subscribe to “Do everything by hand until it’s too painful, then automate” Brian Chesky. Dev time is expensive and code is hard to change. If we’re doing something manually, a) you learn from it, b) you know the actual reqs and c) you can iterate super fast before we make it more concrete in code.
“At the same time, if you do feel that a certain resource is imperative to the project’s success, you should justify how that resource will increase ROI and make a case for it. No need to waste resources but at the same time know that resources will be available where needed and justifiable”
Justina – Xenditer
As you build our your idea, you should also ask yourself, what are the biggest assumptions or foundations for this project? This is also what would make the project fail. This would help you prioritise what hypotheses you need to test so you can guard against such a failure. When we built out remittances from HK->Indonesia we had 3 assumptions: 1) people wanted to send money via an app, 2) people hated waiting in long lines on their 1 day off and 3) Western Union was expensive. We discovered within 3-4 weeks we were wrong and so pivoted away.
How to start?
Ping Moses. We control experiments centrally because most of our attempts have failed (that’s ok). It takes a specific kind of hustle and persistence and luck to make an experiment succeed.
Depending on the case, Moses will likely ask for:
- Create deck / 6 pager – taking into account all the inputs above in CCCP format, success metrics and projected TPV/txn count/revenue impacts
- Demonstrate customer demand – who will actually use this from Day 1
