Multivariate Testing

The attention span on the Web has been decreasing ever since Google had arrived and changed the rules of the game. Now with millions of results available on any topic imaginable, the window to grab a visitor’s attention has decreased significantly (in 2002, the BBC reported it is about 9 seconds)


Statistical testing relies on design of experiments. Several methods in use for multivariate testing include:

1. Discrete choice and what has mutated to become choice modeling is the complex technique that won Daniel McFadden the Nobel Prize in Economics in 2000. Choice modeling models how people make trade offs in the context of a purchase decision. By systematically varying the attributes or content elements, one can quantify their impact on outcome, such as a purchase decision. What is most important are the interaction effects uncovered, which neither the Taguchi methods nor Optimal design solve for.
2. Optimal design involves iterations and waves of testing. Optimal design allows marketers the ability not only to test the maximum number of creative permutations in the shortest period of time but also to take into account relationships, interactions, and constraints across content elements on a website. This allows one to find the optimal solution unencumbered by limitations.
3. Taguchi methods: with multiple variations of content in multiple locations on a website, a large number of combinations need to be statistically tested and medium/low traffic websites can take some time to get a large enough sample of visitors to decide which content gives the best performance. For example, if 3 different images are to be tested in 3 locations, there are 27 combinations to test. Taguchi  orthogonal arrays can be used in the design of experiments in order to reduce the variations but still give statistically valid results on individual content elements. Taguchi uses fractional factorial designs.