Testing Guidelines - Common Mistakes

The results of your A/B-Tests can only be as good as the data you collect. It is therefore of critical importance to set-up your experiments in a way that make efficient use of your traffic and are methodolically sound.

Non-Randomized Group Assingment

The decision which users are assigned to the treatment and control group must be randomized. Non-randomized user assignment may severely impact your test results! Make sure there are no systematic differences between your test-groups - be sure to compare apples with apples!

Example: Instead of randomly assigning users into treatment/control, you accidentally separate them by whether they are returning customers or new ones. Setting up your experiment this way will not yield any information about your new features, but instead will measure the difference between your different customer groups!

Premature User Allocation

Only enlist users into the experiment once they have seen the part of your app that you wish to test. Failure to do so will at best increase the time your test will take (by decreasing efficiency) and at worst will invalidate your experiment!

Example: You want to test whether adding high-resolution images to the product detail page of your webshop leads to an increase in sales. However, instead of enrolling users into the experiment only once they have seen the details page you accidentally enroll all users entering your site. The consequence of this is two-fold: Not only are you counting more users than necessary, you are also potentially counting successful outcomes for users that have seen your new feature (or the old equivalent!)

Mixed Messages (Changing User Experience)

Make sure users don't change groups during the experiment