Re: AMT data
Chris Callison-Burch
This is an excellent idea. I'm forwarding the idea to the workshop mailing list. Folks, please start thinking about this in preparation for the final drafts of your shared-task papers.
--Chris
Begin forwarded message:
> From: Maxine Eskenazi <ma...@cs.cmu.edu>
> Date: March 14, 2010 2:10:58 PM EDT
> To: Mark Dredze <mdr...@cs.jhu.edu>, Chris Callison-Burch <calliso...@jhu.edu>
> Subject: AMT data
>
> Folks,
> After reading the two short papers, I am sending you a suggestion. I have found that the quality control on this work is highly variable. It scares me to think of what scholarly venture the data might be used in to conclude something. If the AM-sponsored data is to be made public, I think there should be some sort of overall quality control. For example, on the page where the download is available, the authors could be asked to display a standard form that might include:
> - a screenshot (or equivalent) of the HIT including the instructions to the workers
> - measures of overall intra-worker agreement
> - comparison to a gold standard (which should be almost universally applicable), with description of how the gold standard was obtained
> - costs, acceptance rate, time it to to get the HIT(s) completed on AMT
> etc.
> Cheers,
> Maxine
>