Mechanical Turk

[Alton Alexander]

Hi, I have a few use cases I'm considering exploring with Orchesta's workflow management and they may involve using Mecanical Turk. Can you describe or point to some use cases for how I can integrate mturk? For example will I just treat that step like an automated machine process and followed by a human reviewer or do you recommend a tighter integration?

[Adam Marcus]

Alton,

Thanks for reaching out!  I'll try to keep this email short, though I welcome you to reach out via chat on gitter or to follow up with any of the options we lay out for more detail.

First, a caveat: we haven't used Orchestra with Mechanical Turk, and in many ways their models don't fully align.  On Orchestra, workers must log into the system and are assigned work through the task dashboard, and so sending them individual tasks via other platforms isn't something we've tried before.  Orchestra's design is also more geared toward having trusted workers review the work of others on relatively long tasks, whereas systems like Mechanical Turk opt for asking several workers to complete the same microtask assignment to ensure quality.

That said, if you're interested in trying Orchestra + Mechanical Turk, here are a few options:

- As you said, you can send/retrieve results using machine tasks.  You could imagine integrating with a system like ampcrowd to support this.  Once the data is returned from the machine task, other workers on the Orchestra platform can further finesse the data.

- If you wish to instead use human tasks rather than machine ones, you have a few directions to go from there.

-> You can create a Mechanical Turk ExternalQuestion that embeds each Orchestra task in an IFrame.  With a small amount of javascript in an Orchestra task, you could capture things like the the assignment/turker ID, and with a submit.success signal, you can capture the user's submission and POST it back to Mechanical Turk for payment.

-> You can also send workers to the task to complete in Orchestra, and generate a key that they then copy/paste into the MTurk task as proof that they have completed the task in Orchestra.

Sorry for the long response, and we're happy to follow up by email or on gitter chat.  If it helps to follow up by hangout/skype, also let us know.  Thanks for your interest!

-Adam

On Wed, Oct 14, 2015 at 2:46 PM Alton Alexander <alexand...@gmail.com> wrote:

Hi, I have a few use cases I'm considering exploring with Orchesta's workflow management and they may involve using Mecanical Turk. Can you describe or point to some use cases for how I can integrate mturk? For example will I just treat that step like an automated machine process and followed by a human reviewer or do you recommend a tighter integration?

--
You received this message because you are subscribed to the Google Groups "Orchestra" group.
To unsubscribe from this group and stop receiving emails from it, send an email to orchestra-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/orchestra-devel/e4f36acd-4e24-433f-baa4-a128e4c33796%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.