crowdlog

Dean Eckles has a very thoughtful post on his blog about how to manage throughput and prioritization of Human Intelligence Tasks (HITs) on Mechanical Turk. It’s something I’ve been thinking a lot about, and so I thought I’d elaborate on some of his points.

Dean breaks down the factors that affect latency and throughput on mTurk:

1. HIT and sub-tasks duration
2. Available workers
3. Appeal of your HITs
4. Reliability

In conversation with Brendan O’Connor from Dolores Labs a few months back, we tried to come up with a simple way of thinking about what affects the likelihood of participation. Building from these thoughts, I think these are the factors:

1. Appeal — How fun the task is
2. Effort — How much effort is involved
3. Reward — How much the payment is
4. Duration — How long the task takes
5. Visibility — How likely your task is to be seen

These are similar to Dean’s, which shows they echo the experience of multiple ‘requesters’ on mTurk. One addition I’d like to suggest is to think of the relationship between factors: a particularly fun task will get good response even if the payment is low, and vice-versa; task length and payment also are related (I’d like to measure this by scaling payment with time and measuring throughput); particularly difficult tasks are often ignored unless they have higher rewards; and, finally, none of this is any good if your task doesn’t get selected by anyone.

This last issue is one I hadn’t understood until I got engaged with the “Turker Nation” community, asking them why our tasks were getting fewer responses than I’d like. The answer was that experienced ‘turkers’ look for three things when choosing a Human Intelligence Task (HIT):

1. A Requester they know and trust (Amazon, one of the better-known transcription services, etc)
2. A HIT with a lot of assignments that they can perform in a row, so they can get in a ‘groove’
3. A recently created HIT

As independent researchers without the brand-name appeal of 1 and often without the tasks that provide for 2, the recency becomes an important factor. Distributing your assignments over a period of time by adding them to the system as interest in your task dies off has been an effective tactic for managing recency; updating your HIT (with any new detail or assignments) will push you to the top of the recency list.