Build, Measure, Learn with Lean Experimentation
The Lean Startup provides a scientific approach for companies to build the right product for customers. The underlying principle of being lean is to validate hypotheses as fast as possible. To validate an idea, you have to carry out experiments.
A scientific product development lifecycle will include lean experimentation during both the discovery and delivery phases. This approach means validation occurs before the feature is built and after it has been deployed.
This augmented version of the Lean Startup's Build, Measure, Learn loop includes running lean experiments during the Build Phase. It will allow for iterating on an idea and failing them as fast and cheaply as possible. Bad ideas are discarded or altered early, and good ideas will move towards product delivery much quicker.
💡 Ideas
Ideas can come from many sections of this loop, such as feedback pages, support forums, in-app feedback tools, your developers, etc. These ideas are best kept in Product Management tools like ProdPad where feedback aggregation and discussions happen. A Project Management tool like Jira is not suitable for this job, but an integration can connect your work across platforms.
Ideas are written as hypotheses that states that a particular change will improve a set of metrics.
eg. "Users want to quickly share their dashboards as an image, which will increase engagement of our app".
These hypotheses will be used to build and carry out experiments that can quickly falsify that hypothesis.
🔨 Build
The build phase can produce 2 types of experiments:
A minimum viable product.
A lean experiment.
The MVP is the typical way we test ideas; this can take weeks to months. With enough thought, a lean experiment can be created to test almost all hypotheses instead of investing in engineering resources to build an MVP.
By using a lean experiment, we can test the previous hypothesis above in many ways. For eg. "setup a fake share button and see how many people click on it. This would tell us if there is interest in the feature.
Product Experiment tools are usually low to no-code targeted to product managers to help set up and run experiments. Tools such as Samelogic or Optimizely are used to conduct the experiments in your application using a wide variety of experiment designs
📐 Measure
All experiments must measure something and must be considered during designing and building the experiment. Measuring is considered as collecting two types of metrics:
Quantitative is primarily a numerical data collection method that answers the "what" or "how".
Qualitative is a form of data that involves more interaction and input from the user that answers the "why".
A rigorous research method will seek to understand both the how and why. A lean experiment tool like Samelogic will allow running an experiment that answers both. This will enable us to iterate on the next idea quickly because we design with the why.
🧪 Lean Experiments
Lean experiments are ways to test hypotheses without the need for writing code.
There are many experiment designs for validating ideas. Some of this include:
Landing Pages
These are graphical and straightforward pages that explain the value proposition of a product or feature. Their main goal is to measure interest by collecting metrics on the call-to-action. We could collect quantitative and qualitative data here, such as popping up a Microsurvey when the CTA is clicked.
Customer Surveys
This process can happen face to face as a product discovery interview. You can also do this digitally, such as sending out surveys or in-app using feedback tools such as Microsurveys.
Doing this in-app allows you to pinpoint the exact user persona to collect contextual feedback, such as a specific user segment, users who have completed particular interactions, etc.
Painted Doors / Fake Doors
The underlying principle of Painted Doors is the same as for Landing Pages; we want to measure the interests of a particular feature. Painted doors are usually simple mocked-up UI elements with no functionality, such as a button or link.
We should collect quantitative and qualitative data here, such as popping up a Microsurvey when the user clicks the painted door.
A/B / Split Testing
The most robust scientific research method for knowing if the introduction of a change is working, there should be randomization controlled comparison between two branches. In product, this is done as A/B testing, where we show the user either an A or a B version and measure the interactions.
This method is primarily quantitative, but we can attach a Microsurvey to either branch to understand why either option worked. An adequately designed split test can tell you the following:
Should I invest in A or B?
Should I throw out both A and B?
Do I need more discovery on A or B?
Can I write an epic/story/task using qualitative data from A or B?
📖 Learning
After every lean experiment, we will need to analyze the results and make decisions based on this data. So not only are we data-driven, but we use robust scientific processes to arrive at and explain that data. This scientific process inspires confidence when taking one of the following actionable outcomes from this process:
Should I move this feature to the team's backlog?
Should I plan capacity around this feature?
Should I iterate on the experiment and test again?
Samelogic is a product that helps Product Teams introduce lean experiments into their workflows without disrupting their current flow.