Fake Door Testing our Figma Integration using LinkedIn
As Product Managers, we must ensure that we are not building things people don't want. In a Product-Led company, our product backlog is constantly validated at the front of the funnel. Before the engineers even get to it.
This is a series of articles that publicly shows all our experiments. We will demonstrate how we validate which integration to build first, such as our Figma integration. You can check how we validated our Amplitude integration here.
How we experiment
Most new features aim to bring in new users or keep existing users happy. In our case, we primarily care about bringing in new users. That means we must run validation experiments at the front of our funnels.
We use a form of an experiment known as fake doors. This test goes by many names; painted door, concept test, and placebo test.
A fake door is when a user interacts with a potential feature or product that is not functional. A mock is placed inside the product to see if users want it by clicking on it. Then we show a survey to get more information from them.
We can test to see if users want a particular integration built by placing a mock of it alongside the rest of your integration.
The azure integration is fake and will show a survey when clicked.
But we are validating for new user demand, so we also have to design these tests at the start of the funnel.
Our funnels
We have tight control over the funnel sources of social network ads such as LinkedIn, Facebook, and Twitter. We can throw out an ad representing the value prop of the feature and see how many people click it, without building anything.
The ad's call to action is to visit whatever landing page we spin up for this feature or use case.
From there, we try to get users to sign up.
Once they sign up, they also have to interact with the fake door in the product, such as showing the intent of using the integration. Iterating on this funnel is also an iteration of the product.
Social Network Ads
When using ads, we measure two main things:
CTR - click-through rate, which is the percentage of people shown who clicked the ad. Our industry standard is 1-2%.
CPC - cost per click, we are essentially buying traffic from our audience, which should be our user personas.
LinkedIn showed the best promise of the three networks. Primarily because LinkedIn has rich career data to filter and fine-tune the ads.
We spend about $70 on each network to get a baseline, and our early metrics showed:
Facebook: 0.2% CTR, $1.85 CPC.
Twitter: 0.1% CTR, $4 CPC.
LinkedIn: 1% CTR, $2 CPC.
This is an average of the data we got from our previous Amplitude experiment.
Setting up the Figma Experiment
To get the Figma funnel test up and running, we set up a landing with a signup button as the CTA, then wrote a post introducing the integration with a link pointing to the landing page.
All the links are customized with unique UTM attributes, so we can track where the traffic is coming from.
We then boost the post to our target audience for two days. LinkedIn gave us the best audience management behavior:
Our spend this time was:
Facebook: $25
Twitter: $20
LinkedIn: $40
Analyzing the results
On a Monday evening, we had 1500 impressions with a 4% CTR and $0.7 CPC on LinkedIn.
At the end of Tuesday, our metrics were:
Facebook: 0.2% CTR, $1.85 CPC.
Twitter: 0.2% CTR, $4 CPC.
LinkedIn: 2.5% CTR, $1 CPC.
This test confirmed interest in the Figma integration, and we should investigate further. So our Amplitude work has now fallen behind.
This test also confirmed that we have a repeatable model using LinkedIn, and we should ignore the rest of the networks. LinkedIn is good enough on its own for idea validation, especially for our type of user personas.
What's next?
We will iterate on some more market validation using Figma for In-Product Concept Testing.
So far, we have bought 40 visitors from our audience at $1 each to visit our landing page. None of them converted, and this shows that we need to revisit our landing page.
Now we are getting into conversion rate optimization, where AB testing tools like google optimize and Optimizely shines.
We will try to get the message across above the fold on the landing page, buy about 40-50 more users and analyze. If we see interest or even a few signups from this funnel test, then that is a strong signal that we should continue investigating the Figma integration.