Efficient Work on MTurk

We created Efficient Work on MTurk, an online learning module through the Open Learning Initiative (OLI) to teach workers on the Amazon Mechanical Turk (MTurk) Platform. We performed cognitive task analysis to identify skills that expert workers use to earn money, iteratively refined our goals, instruction, and assessment, and implemented our course on OLI.

Our goal was to improve the worker experience on Mechanical Turk. Due to the on-demand and remote nature of MTurk, it's easy for workers to be exploited for very low pay. Since many workers use Mechanical turk as a primary and necessary source of income, we wanted to provide training and information for new workers to help them earn more money.

We started by performing cognitive task analysis of how expert workers use the platform by asking them to screen record themselves while narrating their thoughts in the moment. We used this analysis to create goals for our e-learning module. We then created instruction in the form of slides to teach these skills, as well as pre- and post-quizzes to assess their prior knowledge and knowledge gained from our module. We iteratively refined our instruction and assessment as our goals changed.

Finally, we implemented our module using the OLI authoring tools. We used the module to perform an A/B test with 20 participants in each condition to determine if a particular learning science principle had an affect on learning. We then used the pre-post quiz score difference to calculate learning between the different conditions.

After our initial implementation in the fall, in the spring we refined our OLI module for use in in a controlled experiment of MTurk workers to investigate the connection between peer feedback, the skills and knowledge gained in our module, and precise measures of day-to-day productivity on the platform.