Founder Biases #2: Your Product Idea is Wrong

In a conversation with Lisa Grimm from DEMI, she mentioned a story of how their team approached building their product:

“When Ian (Ian Moore, DEMI’s founder) approached me to join DEMI, he already had an engineer building the product and honestly, I thought to myself “What product? We don’t know what we should be building yet!“.

I told him, (gently) that the idea of creating our own product is super exciting but in order to feel confident in what to build we needed to develop hypotheses and validate them by testing and iterating on them as quickly as possible. When I suggested using a third-party platform to test, his concern was that users wouldn’t feel comfortable “joining DEMI” on a non-DEMI app but I challenged him that if that was the case, the product’s value proposition wasn’t compelling enough.

Turns out our value proposition was quite strong, and we tested our core hypotheses for DEMI in record time using WhatsApp which has given us the insights to build a product we couldn’t be more confident in and excited to launch.”

Now, the way DEMI has been approached so far is proper lean-startup: a few core hypotheses, based on a Cause to empower certain communities that the team knew well and was passionate about, a few fast & inexpensive experiments to test those hypotheses, then small pivots to focus the vision (to scale it down), gathering data to help them validate a certain behaviour that was essential to the survival of the product, and only when clear validation started to consolidate, invest in their own MVP.

You’d think that this way of building a product is the norm for product startups. It hardly is, unfortunately. The typical is rather a combination of doing ‘some’ validation, but not too much, as doing too much might just invalidate those initial ideas and really put a dent in our identities as smart founders. The bad news is that this way of doing things trades psychological comfort for market validation and it leads to underperformance in the real world. This was proven in an experiment with 116 Italian startups.

Lean + scientific methods

Arnaldo Camuffo, from Bocconi University in Milan, and a few colleagues conducted an experiment with 116 Italian startups, where they wanted to learn how training half of them on the methods of the scientific approach might influence the performance of their companies.

The 116 startups were split into two groups: a control group and a ‘treatment group’.

“The difference between the groups was that we taught the treatment group to use a more scientific approach throughout the process. First, they learned to use first-principle thinking, which allowed them to identify assumptions and leaps of faith they had made, as they re-examined their business idea. Then, they examined the relationships among the components (“value propositions,” “cost structure” and so on) of their business model canvas and got into the habit of assessing the entire model holistically. They were also trained to collect evidence through robustly designed experiments and rigorous data analysis. Finally, we nudged them to articulate decision rules at the start of their experiments or interviews that would help them to stay the course — or change direction.”

After over a year of research, the results are revealing:

“Over the course of the training and for ten months after, we conducted a total of 16 phone interviews (observations) with each startup. For the 44 that dropped out, interviews were conducted up to the point they exited. Compared to the control group, the treated group had more dropouts (24 vs. 20) and more pivots (19 vs. 11). They also earned more revenue: we recorded 85 positive revenue observations in the treatment group compared to 22 in the control group over a one-year period. Average and median revenue reached €7,800 and €1,300 respectively in the treatment group, versus €900 and €500 in the control.”

In other words: the teams that used the scientific method on top of the typical lean methodology manage to either invalidate their product faster, therefore stop losing unnecessary time and money on a dead horse, or pivot more often into a different direction. Also, the average revenue generated by the treatment group was over 8X compared to the others!

My colleagues and I found that training founders to think like scientists could help reduce the risk of sticking with ideas that don’t ultimately work out. In a randomized controlled trial on 116 early-stage start-ups, we show that entrepreneurs who were taught to formulate hypotheses from theories and rigorously test them on carefully chosen samples of potential customers were more likely to acknowledge that an idea was bad, pivot from non-starters or pitfalls, and generate more revenue than the control group.

The quotes are from the article in HBR but you can also see the original paper here.

A better way

The right Cause & a scientific mindset

But building the product right is just the second phase in a bigger reality. And that reality starts with working on the right Cause. In this game of survival, why you start this journey and for what Cause are way more important than the product you want to build. Because most likely, the first iterations of your product get you, in various degrees, the wrong product. 

It can become less wrong [sic!] by a smarter way to build-launch-test-learn-iterate like proven by Camuffo’s team. But you will still meet many hurdles along the way and if you’re not doing this whole thing for the right reasons you will be tempted to quit. Even the founders who are the most motivated and motivated for the right reasons are tempted to quit, but many don’t because their Cause is higher than the product. The product is just a tool to advance such a Cause, and they will pursue the cause no matter how many times they need to iterate the product in order to get it right (‘less wrong’).

Like in a climbing expedition on Everest, the path to the top has a few milestones and it’s a journey of survival. In almost every startup journey, there is a Death Zone that we need to cross. It’s usually called getting to Product-Market Fit. Then ‘crossing the chasm’. Then it changes to ‘sustainable growth’. Then it switches into ‘responsibility for the Cause’.