Looking good ain’t easy. While we’re working on our new website, feel free to let us know if anything looks amiss by emailing us at brookfield.institute@ryerson.ca.

From Bottom to Top: How Amazon Mechanical Turk disrupts employment as a whole

From Bottom to Top: How Amazon Mechanical Turk disrupts employment as a whole

Crowd workers earn far below minimum wage in many countries. Learn how crowd work is changing employment as we know it via Kristy Milland, as part of our ongoing series on inclusion and equity.
March 14, 2019
Print Page

Kristy Milland is community manager of Turker-Nation.com. She has been a crowd worker for more than a decade, as well as a Requester and a researcher. 

It is easy for people to think their jobs are safe from automation. For a psychologist, the therapeutic alliance between two people is said to be what makes the treatment work. For lawyers, it is difficult to imagine that the creative thinking often required to help their clients find success in the courtroom could be replicated by a computer system. Politicians, artists, writers, accountants, professors, doctors, and others who need education, skill, or inborn talent to do what they do well may expect that their jobs will continue to exist, even when all others are completed by robots or computers. Some feel that their long-term education, that which is required in their industry, will insulate them from replacement. Others think that the creative talent they have been honing since they were a child is irreplaceable. Yet, what keeps them in a position where they can use their experiences and abilities to earn a living wage is that someone values their work highly. What tends to be overlooked is that there is someone else out there who can do what they do—but for less. In fact, there are hundreds of thousands of people in need of money who will use their skills and talents as a crowd in ways that put those in what are deemed to be ‘stable careers’ out of work. All it takes is for economic actors to decide that workers no longer need such an education, or that inborn talent is not worth a fair and decent wage. What follows next is that these jobs can then be given to individuals who are willing to work for far less. It is this combination of automation and a crowd of willing workers, which may lead to the decimation of white collar work as we know it, unless we confront the future of work head-on.

Of course, this sort of move cannot happen if there is no one to give the work to. There are dozens of platforms that offer large crowds of workers willing to do tasks for pennies a piece, such as Figure Eight, One Space, Clickworker, and Microworker. The oldest and most well known of them all, Amazon Mechanical Turk (mTurk), not only facilitates the work itself through its online labour platform, but has gathered a crowd of more than 100,000 workers willing to do anything and everything for money (Difallah, Filatova, & Ipeirotis, 2018). You can imagine the site to be like a bulletin board you might find at a community job centre, or even in your local coffee shop. In such an offline environment, people in the area could print off a job ad explaining what they would like done, and workers nearby could check to see if there is work they can do. For example, if I wanted to translate an article I had written from English to French, I might stick it up on that board with the price that I am willing to pay and my contact information. Then someone who knows both languages well would take the article home, work on it, and bring it back to me in exchange for the payment promised. It might take a few days or even a few weeks, depending on the complexity and length of the text. I might end up paying a few hundred dollars to ensure the best work was received. Yet, if I am not willing to wait so long or pay so much, I could instead put that article up on mTurk. First, I would cut each sentence out and submit each individually; this is a system of posting work called “microtasking”. Then, once they were all posted on the site, anyone who knows both languages would grab as many sentences as possible and do them in tandem, meaning a 1,000 sentence article could be translated in as little as six minutes. That whole article would likely only cost me $20 to translate. mTurk makes getting work done cheap, easy, and quick. Thus, there is no reason why, as AI learns to do the jobs we need done, companies will not start moving jobs to mTurk or other platforms.

As a worker on mTurk for the last 13 years, I have probably seen it all. From writing titles for porn videos to tagging ISIS video screenshots to categorizing images and videos based on their violations of content moderation policies for Fortune 500 companies, it has not always been enjoyable. Yet, one type of work which has increased in availability on mTurk over the last eight years has been that which clearly powers automation, robotization, and AI. Crowd work platforms, it turns out, are now key to technological innovation in the digital realm. For example, mTurk workers create datasets that are used to train AI algorithms. We are a part of the process of training AI from beginning to end. First, we create gold standard datasets; essentially, we provide the correct answers to questions that are to be posed to AI. For example, we might take 10,000 photos stating whether they include a cat or a dog. These answers are then fed to algorithms to teach them to determine the difference between cats and dogs in images. Next, the algorithm tries to figure it out on its own, and we adjudicate the results of its work, which is then fed back to the algorithm to fix any flaws it may have. Lastly, we often judge the algorithm on its “human nature” – does this seem like a robot completed the work, or a human? Many companies want their AI to appear humanlike, such as a chatbot for customer service, so we are tasked with making it more like us.



The drive to reduce the cost of labour to zero is fueling the development of AI, automation, and robots, yet until those technologies are a reality, companies are employing workers for pennies a task. And, the fact that wages have not kept up with production, leaving people without enough income to survive, is pushing workers to do those tasks.

Our work does not end at AI; we train and judge robots as well. From taking videos of ourselves doing a task so that the robot can mimic our movements, to deciding if a robot is using the most efficient movements, we train them much as we do AI. We also help with visual recognition, one of the most complex jobs of a robot. We judge whether a robot guessed right about something it ‘saw’ or process images in order to give it a database of answers to draw from. We do the same for audio input, such as voice recognition. We supply our own voice in audio files, or transcribe the voice input of others. Not all of these tasks are so mundane and innocent though, as we also train drones. For example, one task came with instructions that showed three black and white stills from a video. Shot from above, it showed a man walking away from a car on what appeared to be a large dirt field. The instructions said to identify from this image what direction the man appeared to be going in. Once you read the instructions, you were given another image of a person standing on a dirt area and asked which direction they were walking in. Why would a drone need to know this? This i because when you shoot at someone, you do not shoot at where they were standing when you fired, rather where they will be when the bullet reaches them. To do so, you have to know which direction they are walking toward.

Research, creation of data sets, data processing, analysis of results, content moderation, training chatbots, sensory recognition – these are all tasks that, at the moment, only a human can do. Yet, these are also tasks one would imagine being done by employees in tech companies who are compensated fairly for their efforts. Few realize that these tasks are instead going to a crowd of workers, invisible to the consumers of this data, for a median wage of $2 USD per hour (Hara et al., 2018). What may also be shocking is that a number of workers on mTurk are from the United States (Ipeirotis, 2010). In a country where the cost of living is so high, and Federal minimum wage is set at $7.25 USD per hour, one might wonder how this sort of platform can be permitted to exist. The same is true of Canada, where over 300 mTurk workers responded to a survey I posted recently. Labour law does not cover independent contractors in either country, which is what Amazon deems its workers to be, so there is no floor to how little they can be paid, how much they can work, or what sort of work they can be tasked with. With no controls, mTurk workers can replace any worker at a fraction of the cost and many times the speed.

This work does not have to be so destructive for those of us in the crowd, and one method of ensuring ethical treatment is to allow the workers to run the platform themselves. Cooperative businesses have been around for centuries, where the workers make the decisions and control the organization they work for. Successful examples include Agropur in Canada, Mondragon in Spain, and Land O’Lakes in the United States. The Platform Cooperativism movement, which takes its name from a term coined by Trebor Scholz at The New School in New York, seeks to apply cooperative values and experience to creating online platforms run by workers. For example, Fairmundo, a competitor to Amazon, is a cooperative where members take a share of the profits; and Stocksy, a photographer coop, allows members to co-own the platform, having a say in decision-making. These platforms are proof-of-concept that individual stakeholders can work together to create a site, one which even competes with the biggest names in the online industry. That means there is nothing stopping workers from banding together, creating a website, and offering their work on their terms. With government and non-profit business loans and grants, this reality could come sooner than one might think. In fact, recently Google.org provided a $1 million grant to Scholz’ Platform Cooperative Consortium in order to build a toolkit that allows anyone to create a cooperative platform easily and quickly (OCAD, 2018).


If we do not want the future to be crowd work for all, we have to support more ethical alternatives, or a push for the government to begin regulating independent contractor status, in order to stop its progression.

There is never just one solution to a problem, and thus it is important that powerful actors in the economy also are on board with helping protect crowd workers and freelancers from exploitation. For example, labour legislation does not currently cover freelancers or independent contractors in Canada. This means that the benefits that employees take for granted, such as sick days, vacation time, protection from unjust termination or wage theft, constraints on working hours, and minimum wage, are unavailable to anyone classified as an independent contractor. We must modernize our laws so that even those who take on gig work can be promised a fair and equitable work environment, as their counterparts in traditional employment relationships already enjoy. In addition, specific new legislation would be of benefit. The Independent Drivers’ Guild in New York was founded based on their ability to access a mailing list of drivers, thanks to an agreement with Uber (Johnston & Land-Kazlauskas, 2018). Legislation that requires all companies to connect their freelancers to organizations interested in helping workers improving their working conditions would engender massive change in industries, such as crowd work, where the majority of workers never see each other, let alone communicate about their workplace. There are likely many other laws which could help such workers have a better work experience, but until we push legislators to take a serious look at the conditions of this work and the problems those conditions are creating, it is unlikely that we will be able to stop this race to the bottom.

As a worker I have found that nothing in the world of crowd work is easy. You struggle to make an income that is enough to feed your family, your work is precarious and could disappear at any moment, and being tied to a computer to not miss out on new work postings can be ruinous to your personal life. Creating a cooperative platform and competing with conglomerates such as Amazon is also going to take a lot of effort. But anything that improves the dire labour conditions and wages of mTurk workers would be an improvement. The fact that this exploitive crowd work exists is an indication that our economy is on a problematic trajectory. The drive to reduce the cost of labour to zero is fueling the development of AI, automation, and robots, yet until those technologies are a reality, companies are employing workers for pennies a task. And, the fact that wages have not kept up with production, leaving people without enough income to survive, is pushing workers to do those tasks. If we do not want the future to be crowd work for all, we have to support more ethical alternatives, or a push for the government to begin regulating independent contractor status, in order to stop its progression. We must protect and create good quality jobs and labour conditions for workers across the economy that are insulated from the entwined impacts of automation and gig work.


Chiang, C., Kasunic, A., & Savage, S. (2018,  November). Crowd Coach: Peer Coaching for Crowd Workers’ Skill Growth. In Proceedings of the ACM on Human-Computer Interaction, Vol. 2, No. CSCW, Article 37.

Difallah, D., Filatova, E., & Ipeirotis, P. (2018). Demographics and Dynamics of Mechanical Turk Workers. In The Proceedings of WSDM 2018: The Eleventh ACM International Conference on Web Search and Data Mining. Retrieved from http://www.ipeirotis.com/?publication=demographics-and-dynamics-of-mechanical-turk-workers

Government of Canada. (2018). What we heard: Modernizing federal labour standards. Retrieved from https://www.canada.ca/en/employment-social-development/services/labour-standards/reports/modernizing-federal-standards.html

Hara, K., Adams, A., Milland, K., Savage, S., Callison-Burch, C., & Bigham, J. P. (2018, April). A Data-Driven Analysis of Workers’ Earnings on Amazon Mechanical Turk. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (p. 449). ACM: New York.

Ipeirotis, P. G. (2010). Analyzing the Amazon Mechanical Turk marketplace. XRDS: Crossroads, The ACM Magazine for Students, 17(2), 16-21.

Johnston, H., & Land-Kazlauskas, C. (2018). Organizing on-demand: Representation, voice, and collective bargaining in the gig economy (CONDITIONS OF WORK AND EMPLOYMENT SERIES No. 94). Retrieved from https://www.ilo.org/wcmsp5/groups/public/—ed_protect/—protrav/—travail/documents/publication/wcms_624286.pdf

Kessler, S. (2018, June 5). The Unequal Geography of the Gig Economy. The Atlantic. Retrieved from https://www.theatlantic.com/business/archive/2018/06/gig-economy-inequality/560942/

OCAD. (2018, June 1). Initiative co-led by IDRC and The New School receives $1M Google grant. Retrieved from https://www2.ocadu.ca/news/initiative-co-led-by-idrc-and-the-new-school-receives-1m-google-grant

Ontario Ministry of Labour. (2017). The Changing Workplaces Review. Retrieved from https://ontario.ca/document/changing-workplaces-review-final-report

For media enquiries, please contact Coralie D’Souza, Director of Communications, Events + Community Relations at the Brookfield Institute for Innovation + Entrepreneurship.