Doing 1 task with multiple annotators

[wajdi zaghouani]

Hi all,

I have a list of 100 words in a csv file and I want to get the part of
speech for each word done by 6 different annotators. I am using the
web interface to build the hit.

How can I ensure that the 100*6 hits will only be done by 6 people and
not divided randomlly. for instance 10 people sharing the 600
annotation instead of 6 people sharing the 600 annotation?

Thank you for your help,

Wajdi

[Brendan O'Connor]

i think the only way to do this on MTurk is by putting all of the work
into one giant HIT.

--
http://anyall.org

[Chris Callison-Burch]

When you design a new HIT you can specify the "Number of assignments per HIT". If you set that to 6, then each word will be assigned to 6 different people. There won't be a case where one person does the same word twice.

--Chris

[Bart Mellebeek]

Mind that this setting ensures a number of unique workers _per HIT_
only. So if you upload 100 different HITs with the value of 'assignments
per HIT' set at 6, in theory the result could be that 600 different
workers do one HIT each. I think the only way to ensure a consistency of
annotators is to put all your annotations into one big HIT, and restrict
that single HIT to 6 different workers.

Bart

[Scott Novotney]

Another (probably unsuccessful approach) would be to create six batches of one hundred hits, each associated with six separate qualifications, which can be assigned to at most one person.

You could do this for N pairs of (unique qualifications , 100 hits) and wait until six turkers complete all 100.

I think you'd need to dip into the MTurk API in order to accomplish this.

-Scott

[Chris Callison-Burch]

This also breaks the mentality of Mechanical Turk, which is to have lots of people do the work. --CCB

[wajdi zaghouani]

Thank you all,

i think I will go ahead and do one single giant hit with 100 entries.

Wajdi

On Mar 2, 6:49 pm, Chris Callison-Burch <c...@cs.jhu.edu> wrote:
> This also breaks the mentality of Mechanical Turk, which is to have lots of people do the work.  --CCB
>
> On Mar 2, 2010, at 6:43 PM, Scott Novotney wrote:
>
> > Another (probably unsuccessful approach) would be to create six batches of one hundred hits, each associated with six separate qualifications, which can be assigned to at most one person.
>
> > You could do this for N pairs of (unique qualifications , 100 hits) and wait until six turkers complete all 100.
>
> > I think you'd need to dip into the MTurk API in order to accomplish this.
>
> > -Scott
>

[Chris Callison-Burch]

I doubt that will work. Why do you need to design your experiment in that way? You should consider re-designing it. --CCB

[wajdi zaghouani]

I want to be able to compare the F-measure of the annoators against a
gold standard annotation and this can only be done by having 6 people
do the same set of data.