In this article we continue our examination of the AGILE statistical approach to AB testing with a more in-depth look into futility stopping, or stopping early for lack of positive effect (lack of superiority). We’ll cover why such rules are helpful and how they help boost the ROI of A/B testing, why a rigorous statistical rule is required in order to stop early when results are unpromising or negative and how it works in practice. We’re reviewing this from the standpoint of the AGILE me...| Blog for Web Analytics, Statistics and Data-Driven Internet Marketing | Analy...
The concept of statistical significance is central to planning, executing and evaluating A/B (and multivariate) tests, but at the same time it is the most misunderstood and misused statistical tool in internet marketing, conversion optimization, landing page optimization, and user testing.| Blog for Web Analytics, Statistics and Data-Driven Internet Marketing | Analy...
I got a question today about our AGILE A/B testing calculator and the statistics behind it and realized that I’m yet to write a dedicated post explaining the efficiency gains from using the method in more detail. This despite the fact that these speed gains are clearly communicated and verified through simulation results presented in our AGILE statistical method white paper [1].| Blog for Web Analytics, Statistics and Data-Driven Internet Marketing | Analy...
After many months of statistical research and development we are happy to announce two major releases that we believe have the potential to reshape statistical practice in the area of A/B testing by substantially increasing the accuracy, efficiency and ultimately return on investment of all kinds of A/B testing efforts in online marketing: a free white paper and a statistical calculator for A/B testing practitioners. In this post we’ll cover briefly the need for a new method, some highligh...| Blog for Web Analytics, Statistics and Data-Driven Internet Marketing | Analy...
The question of whether one should run A/B tests (a.k.a online controlled experiments) using one-tailed versus two-tailed tests of significance was something I didn’t even consider important, as I thought the answer (one-tailed) was so self-evident that no discussion was necessary.| Blog for Web Analytics, Statistics and Data-Driven Internet Marketing | Analy...
Running shorter tests is key to improving the efficiency of experimentation as it translates to smaller direct losses from testing inferior experiences and also less unrealized revenue due to late implementation of superior ones.| Blog for Web Analytics, Statistics and Data-Driven Internet Marketing | Analy...