What should you do when your sample doesn’t match the known population composition on key variables like prior experience? One approach is to weight your data to rebalance the sample. In a previous article, we discussed how to weight means (such as from rating scales) when there are differences between group proportions in a sample […]| MeasuringU
Mobile apps are different from websites. People have different expectations for a mobile app and how it can integrate with their phone and data. While the mobile app experience is similar in many ways to other interfaces such as websites and software, mobile apps are distinct enough that we feel they deserve their own questionnaire. […]| MeasuringU
In A/B testing sequential tests are gradually becoming the norm due to the increased efficiency and flexibility that they grant practitioners. In most practical scenarios sequential tests offer a balance of risks and rewards superior to that of an equivalent fixed sample test. Sequential monitoring achieves this superiority by trading statistical power for the ability to stop earlier on average under any true value of the primary metric.| Blog for Web Analytics, Statistics and Data-Driven Internet Marketing | Analy...
“What sample size do I need?” We’ve all been trained from years of math education to expect a single answer to that question—a single sample size number. But earlier, we warned against the quixotic quest to identify the one true sample size to use for UX research—the “magic number.” Because sampling error is real but […]| MeasuringU
What is Statistical Power?| Blog for Web Analytics, Statistics and Data-Driven Internet Marketing | Analy...
I got a question today about our AGILE A/B testing calculator and the statistics behind it and realized that I’m yet to write a dedicated post explaining the efficiency gains from using the method in more detail. This despite the fact that these speed gains are clearly communicated and verified through simulation results presented in our AGILE statistical method white paper [1].| Blog for Web Analytics, Statistics and Data-Driven Internet Marketing | Analy...
Sample size estimation is an important part of study planning. If the sample size is too small, the study will be underpowered, meaning it will be incapable of detecting sufficiently small differences as statistically significant. If the sample size is too large, the study will be inefficient and cost more than necessary.| MeasuringU