Daily Research News Online no. 4588 - Forget Targeted Samples and Ask the Crowd

Forget Targeted Samples and Ask the Crowd

September 21 2005

In his award-winning ESOMAR paper, John Kearon, CEO & Chief Juicer of BrainJuicer, explains his attempts to apply academic thinking on 'research markets' and the 'wisdom of crowds' to concept testing. The results, while still inconclusive, threaten conventional MR techniques - suggesting that a diverse, non-target group can predict success as well as a target sample.

Kearon's paper 'A Fresh Approach to Concept Testing: How to Get More Research for Less Time and Money' was inspired by the theories of James Surowiecki's best-selling US book 'The Wisdom of Crowds'. The book challenges modern corporate decision-making models, arguing that we should 'stop hunting the expert and ask the crowd instead', stating that under certain circumstances, crowds make better decisions than even informed individuals. For a crowd to be wise, he states, it must be diverse, independent and faithfully aggregated.

Surowiecki raises the possibility of using 'decision markets' or what Kearon calls 'research markets' to predict and make decisions. One successful example is the Iowa Electronic Market (IEM), where betting on the outcome of elections has proved significantly more accurate than conventional political polling. The IEM's election-eve predictions in presidential elections between 1988 and 2000 were, on average, off by just 1.37%. Similarly, the Hollywood Exchange web site - again based on betting - managed to predict 40 out of 40 Oscar winners - more than any newspaper pundit.

Kearon says that applying such 'wisdom of crowds' thinking to MR would 'reinforce the wisdom of using large, quantitative samples for decision-making' - but would of course also throw a huge question mark over the necessity of using targeted samples to predict in-market performance.

His first experiment, carried out in January for Unilever, consisted of comparing the results of two pieces of research:

An online 'research market', using a diverse, non-representative panel of 500 people. Each was asked to state which of the 10 new product ideas they believed would be most and least successful in the market place. The final results aggregated 'favourite' and 'least favourite' choices to mimic the actions of buyers and sellers in a market.
vs
An online standard concept test for each one of the same 10 new product ideas, using matched, representative samples of 100 respondents in the target audience.

Both tests rated the same five ideas as above average - with more extreme, polarised results for the 'research market' - in line with what one would expect from a standard forced choice test. The company then removed the betting mechanism, which it suspected was encouraging participants to second-guess other people's answers and distort the market. This time, the results were less extreme, and produced an even stronger match with the standard product test results. That the 'research market' and a diverse crowd of people got it almost perfectly right surprised Kearon as much as everyone else: 'This should not have happened - remember, this is a crowd of people rather than a representative sample of the target market and a forced choice test which is known to produce more extreme results'.

Kearon is now keen to complete ten similar experiments to test the theories further, testing the minimum number of respondents needed, the optimum number of ideas for respondents to choose from, and the minimum amount of knowledge required by the 'crowd'.

If validated, he points out that the 'research markets' method would be not only cheaper and easier to run but would 'produce an accurate but even clearer result, as the slightly more extreme ratings give more clarity to the best ideas and are more punishing on the weak ones'.

The full paper is available at: www.mrweb.com/drno/brainjuicerpres.pdf.

All articles 2006-23 written and edited by Mel Crowther and/or Nick Thomas, 2024- by Nick Thomas, unless otherwise stated.