Launch new products without experiment

Question

How can you launch a new product if you can't run an experiment? Or how can you adapt a metric so you can run an experiment?

Example in this link: https://hbr.org/2018/11/using-experiments-to-launch-new-products

Uber wanted to launch Express Pool, so they do the tipical A/B testing and compare metrics ,but in this case they have metrics to compare before and after launching the product (revenue per user, avg trips per users, etc.)

But what if this is a complete new product? Example: Uber trying to launch a Wallet?

If I don't have a counterfactual, what can I do?

Answer 1

There are multiple things you can do before launching a new product.

1. You can run survey for different groups of users asking the about their needs. The scope of this survey is to identify problems that you can then solve using your product. This is a very early age tactic you can use to determine if your product has a potential fit into the market.

2. You can create pitches and crowd-founding campaigns The scope of these are to determine if there is a potential demand for your solution in the market. You are basically starting to sell in the idea phase, before even getting to build anything. Note that you don't want to scam people here, you are just trying to determine if there is a potential in the market.

3. You can launch an alpha or beta pre-release version of the product. The scope of this pre-release is to invite few users into your early application and get their feedback. Based on the feedback you get here you can either improve/ change or update your product before launching it.

4. You can launch an MVP (minimum viable product) and then track KPIs in the real world. The MVP can give you enough information so you know where to go. Just make sure that you are tracking the right KPIs Good luck!

Answer 2

You don't always need to come up with specific hypothesis and validate it. Sometimes it's best to understand and quantify how a new feature affects the overall health of your product. A lot of times in the past when we expose a new feature to a small population of users, we'd quickly figure out if everything is working as expected or if there are unforeseen consequences.

This is hard to do without the right tooling. One such tool that provides a holistic view of the product's health is Statsig. Here's a quick screenshot of what to expect when you build and rollout new features (without having to set up a formal A/B experiment).

Disclaimer: I work at Statsig