Crowdsourced consumer data: how do we make sure it's good?

When Amazon launched Mechanical Turk (MTurk) in 2005, executives touted it as a way to augment artificial intelligence with the old-fashioned human variety. Organizations would post details about a small task that needed to be completed, such as writing product descriptions or identifying performers on music CDs, and then people searching the MTurk site would browse the jobs available and start working on those they were qualified for and thought sounded interesting or lucrative.

But as is often the case with an innovation, one of MTurk’s most popular applications seems to have caught Amazon by surprise: social science research. In consumer behaviour research alone, over 15,000 studies have been published based on evidence collected from MTurk workers, making it perhaps the most represented pool of participants in the history of my discipline. In the Journal of Consumer Research, one of the major journals in my area of specialism, 43 per cent of behavioural studies in the June 2015-April 2016 volume were conducted using MTurk.

MTurk and analogous platforms such as Prolific have enabled my colleagues and me to collect samples much more quickly and more cheaply than through traditional alternatives (eg, university participant pools), and those of us who looked into the quality of the resulting data found it to be comparable to such alternatives.

But every powerful new tool creates a new set of risks, and crowdsourced data is no exception. Some fear that the people filling out the materials might not be giving them their full attention. Others worry that people who taking many surveys may no longer be naïve respondents, possibly compromising the results. Finally, how do you know the person filling out the survey is who he says he is? After all, as the old New Yorker cartoon put it, on the internet, nobody knows you’re a dog.

Other issues

Other issues may take more time to work out. Crowdsourced data has been around for roughly a decade now but protocols for its use are still being worked out. Our work has proposed some guidelines for handling it better, but there are other issues that remain. 

For instance, we need time to know the distinctive qualities of the crowd behind the data. For example, one survey has found that American MTurk workers tend to score higher on reporting if they needed to think about an answer (need for cognition) and higher on civics questions. They tend to be younger and better educated than the general run of people. They are also unusual in that they are slightly more introverted, show greater levels of social anxiety, and have slightly lower self-esteem than the general population. This should serve to remind us that absent more sophisticated recruitment tools, we should always treat crowdsourced samples as non-representative.

Also, the technology itself still has plenty of room for improvement. For example, though third parties can help, these sites don’t have a good way yet to handle interaction between participants. Similarly, easier tools to share projects and data across researchers would be helpful. By enabling researchers to collect larger samples, crowdsourcing is already contributing to making consumer research better, but more can be done to facilitate more open collaboration between scientists.

Crowdsourcing websites like MTurk make survey and experimental investigations more efficient. When used conscientiously, crowdsourcing can also help improve consumer science by enabling more numerous and informative studies and increasing participant and researcher diversity. However, online research and crowdsourcing in particular have their own set of risks, and researchers need to design studies in ways that mitigate them.