Background I'm new to running AB tests and marketing experiments in general. I am hoping to do a 50/50 split test for a large consumer app with a weekly newsletter:
- control group (receives no newsletter)
- experiment group (receives a weekly community newsletter with app highlights, events etc.)
I believe those that receive the weekly newsletter, and those that receive and open the newsletter will be more likely to retain in the app. I have been able to analyze retention between the two groups by join date.
Results
- Significant increase in retention between send+open vs. control
- No significant difference in retention between send vs. control
Is it even good science to correlate retention between these groups given there is so much variation that can happen between users outside of just receiving or not receiving the weekly email? (e.g. different intent, super user vs. casual user)