In this article we continue our examination of the AGILE statistical approach to AB testing with a more in-depth look into futility stopping, or stopping early for lack of positive effect (lack of superiority). We’ll cover why such rules are helpful and how they help boost the ROI of A/B testing, why a rigorous statistical rule is required in order to stop early when results are unpromising or negative and how it works in practice. We’re reviewing this from the standpoint of the AGILE me...| Blog for Web Analytics, Statistics and Data-Driven Internet Marketing | Analy...
Learn the meaning of Risk-Reward Analysis in the context of A/B testing, a.k.a. online controlled experiments and conversion rate optimization. Detailed definition of Risk-Reward Analysis, related reading, examples. Glossary of split testing terms.| www.analytics-toolkit.com
Learn the meaning of Generalizability (a.k.a. external validity, representativeness) in the context of A/B testing, a.k.a. online controlled experiments and conversion rate optimization. Detailed definition of Generalizability, related reading, examples. Glossary of split testing terms.| www.analytics-toolkit.com
One topic has surfaced in my ten years of developing statistical tools, consulting, and participating in discussions and conversations with CRO & A/B testing practitioners as causing the most confusion and that is statistical power and the related concept of minimum detectable effect (MDE). Some myths were previously dispelled in “Underpowered A/B tests – confusions, myths, and reality”, “A comprehensive guide to observed power (post hoc power)”, and other works. Yet others remain.| Blog for Web Analytics, Statistics and Data-Driven Internet Marketing | Analy...
How long does a typical A/B test run for? What percentage of A/B tests result in a ‘winner’? What is the average lift achieved in online controlled experiments? How good are top conversion rate optimization specialists at coming up with impactful interventions for websites and mobile apps?| Blog for Web Analytics, Statistics and Data-Driven Internet Marketing | Analy...
