Characterizing the Global Crowd Workforce: A Cross-Country Comparison of Crowdworker Demographics

Lisa Posch; Arnim Bleier; Fabian Flöck; Clemens M. Lechner; Katharina Kinder-Kurlanda; Denis Helic; Markus Strohmaier

doi:10.15346/hc.v9i1.106

Authors

Lisa Posch Graz University of Technology
Arnim Bleier GESIS - Leibniz Institute for the Social Sciences
Fabian Flöck GESIS - Leibniz Institute for the Social Sciences
Clemens M. Lechner GESIS - Leibniz Institute for the Social Sciences
Katharina Kinder-Kurlanda University of Klagenfurt
Denis Helic Graz University of Technology
Markus Strohmaier University of Mannheim, GESIS - Leibniz Institute for the Social Sciences, Complexity Science Hub Vienna

DOI:

https://doi.org/10.15346/hc.v9i1.106

Keywords:

Crowdworkers, Micro Tasks, Demographics, Cross-National Comparison

Abstract

Since its emergence roughly a decade ago, microtask crowdsourcing has been attracting a heterogeneous set of workers from all over the globe. This paper sets out to explore the characteristics of the international crowd workforce and offers a cross-national comparison of crowdworker populations from ten countries. We provide an analysis and comparison of demographic characteristics and shed light on the significance of microtask income for workers situated in different national contexts. With over 11,000 individual responses, this study is the first large-scale country-level analysis of the characteristics of workers on the platform Appen (formerly CrowdFlower and Figure Eight), one of the two platforms dominating the microtask market. We find large differences between the characteristics of the crowd workforces of different countries, both regarding demography and regarding the importance of microtask income for workers. Furthermore, we find that the composition of the workforce in the ten countries was largely stable across samples taken at different points in time.

References

Amazon Mechanical Turk (2016). Amazon Mechanical Turk: Worker Web Site FAQs. http://www.mturk.com/mturk/help?helpPage=

worker#how_paid. (2016). Accessed: 2016-12-20.

Berg, J. (2016). Income security in the on-demand economy: findings and policy lessons from a survey of crowdworkers. (2016).

Berinsky, A. J, Huber, G. A, and Lenz, G. S. (2012). Evaluating online labor markets for experimental research: Amazon. com’s

Mechanical Turk. Political Analysis 20, 3 (2012), 351–368.

Bloomberg News (2016). Venezuela’s Currency Is Collapsing on the Black Market Again. http://www.bloomberg.com/news/articles/

-11-01/venezuela-s-currency-is-collapsing-on-the-black-market-again. (2016). Accessed: 2016-12-20.

Brabham, D. C. (2010). MOVING THE CROWDAT THREADLESS. Information, Communication & Society 13, 8 (2010), 1122–1145.

DOI:http://dx.doi.org/10.1080/13691181003624090

Brewer, R, Morris, M. R, and Piper, A. M. (2016). Why would anybody do this?: Understanding older adults’ motivations and challenges in crowd work. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 2246–2257.

Buhrmester, M, Kwang, T, and Gosling, S. D. (2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on psychological science 6, 1 (2011), 3–5.

Destatis. (2013). Einkommens- und Verbrauchsstichprobe: Einnahmen und Ausgaben privater Haushalte. Fachserie 15 des Statistischen Bundesamtes 4 (2013).

Difallah, D, Filatova, E, and Ipeirotis, P. (2018). Demographics and dynamics of mechanical turk workers. In Proceedings of the

Eleventh ACM International Conference on Web Search and Data Mining. ACM, 135–143.

European Agency for Health and Safety at Work. (2015). The future of work: crowdsourcing. https://osha.europa.eu/en/

tools-and-publications/publications/future-work-crowdsourcing/view. (2015). Accessed: 2018-10-20.

Goodman, J. K and Paolacci, G. (2017). Crowdsourcing consumer research. Journal of Consumer Research 44, 1 (2017), 196–210.

Hirth, M, Hoßfeld, T, and Tran-Gia, P. (2011). Human cloud as emerging Internet application-anatomy of the microworkers crowdsourcing platform. University of Wurzburg Institute of Computer Science Research Report Series (2011).

Huff, C and Tingley, D. (2015). “Who are these people?” Evaluating the demographic characteristics and political preferences of MTurk survey respondents. Research & Politics 2, 3 (2015), 2053168015604648.

Ipeirotis, P. G. (2010)a. Analyzing the Amazon Mechanical Turk marketplace. ACM Crossroads 17, 2 (2010), 16–21. DOI:http:

//dx.doi.org/10.1145/1869086.1869094

Ipeirotis, P. G. (2010)b. Demographics of mechanical turk. CeDER Working Papers (2010).

Kazai, G, Kamps, J, and Milic-Frayling, N. (2012). The face of quality in crowdsourcing relevance labels: Demographics, personality and labeling accuracy. In Proceedings of the 21st ACM international conference on Information and knowledge management. ACM, 2583–2586.

Kuek, S. C, Paradi-Guilford, C, Fayomi, T, Imaizumi, S, Ipeirotis, P, Pina, P, Singh, M, and others, . (2015). The global opportunity in online outsourcing. Technical Report. The World Bank.

Lin, J. (1991). Divergence measures based on the Shannon entropy. IEEE Transactions on Information theory 37, 1 (1991), 145–151.

Martin, D, Carpendale, S, Gupta, N, Hoßfeld, T, Naderi, B, Redi, J, Siahaan, E, and Wechsung, I. (2017). Understanding the Crowd: Ethical and Practical Matters in the Academic Use of Crowdsourcing. In Evaluation in the Crowd. Crowdsourcing and Human- Centered Experiments. Springer, 27–69.

Naderi, B. (2018). Who are the Crowdworkers? In Motivation of Workers on Microtask Crowdsourcing Platforms. Springer, 17–27.

Paolacci, G and Chandler, J. (2014). Inside the Turk: Understanding Mechanical Turk as a participant pool. Current Directions in Psychological Science 23, 3 (2014), 184–188.

Paolacci, G, Chandler, J, and Ipeirotis, P. G. (2010). Running experiments on amazon mechanical turk. Judgment and Decision making 5, 5 (2010), 411–419.

Pavlick, E, Post, M, Irvine, A, Kachaev, D, and Callison-Burch, C. (2014). The language demographics of amazon mechanical turk.

Transactions of the Association for Computational Linguistics 2 (2014), 79–92.

Peer, E, Brandimarte, L, Samat, S, and Acquisti, A. (2017). Beyond the Turk: Alternative platforms for crowdsourcing behavioral

research. Journal of Experimental Social Psychology 70 (2017), 153–163.

Posch, L, Bleier, A, and Strohmaier, M. (2017). Measuring Motivations of Crowdworkers: The Multidimensional Crowdworker Moti-vation Scale. CoRRhttps://arxiv.org/abs/1702.01661 (2017).

Ross, J, Irani, L, Silberman, M. S, Zaldivar, A, and Tomlinson, B. (2010). Who are the crowdworkers?: shifting demographics in mechanical turk. In Proceedings of the 28th International Conference on Human Factors in Computing Systems, CHI 2010, Extended Abstracts Volume, Atlanta, Georgia, USA, April 10-15, 2010, Elizabeth D. Mynatt, Don Schoner, Geraldine Fitzpatrick, Scott E. Hudson, W. Keith Edwards, and Tom Rodden (Eds.). ACM, 2863–2872. DOI:http://dx.doi.org/10.1145/1753846.1753873

Ross, J, Zaldivar, A, Irani, L, and Tomlinson, B. (2009). Who are the turkers? worker demographics in amazon mechanical turk.

Department of Informatics, University of California, Irvine, USA, Tech. Rep (2009).

Shapiro, D. N, Chandler, J, and Mueller, P. A. (2013). Using Mechanical Turk to study clinical populations. Clinical Psychological Science 1, 2 (2013), 213–220.

United Nations Department Of Economic And Social Affairs - Statistics Division. (2000). Classifications of expenditure according to purpose. https://unstats.un.org/unsd/publication/SeriesM/SeriesM_84E.pdf. (2000). Accessed: 2018-12-02.

U.S. Bureau of Labor Statistics (2018). Consumer Expenditure Surveys - CE Survey Materials. https://www.bls.gov/cex/

csxsurveyforms.htm. (2018). Accessed: 2018-12-02.

Weinberg, J. D, Freese, J, and McElhattan, D. (2014). Comparing Data Characteristics and Results of an Online Factorial Survey

between a Population-based and a Crowdsource-recruited Sample. Sociological Science 1 (2014).