NAACL Workshop: Amazon Mechanical Turk: March 3 deadline
Mark Dredze
Below is the final CFP. We have extended the deadline to Wednesday
March 3 at 11:59PM.
Best,
Mark
NAACL-HLT 2010 Workshop on Amazon Mechanical Turk
June 5 or 6, Los Angeles, CA
Workshop Website: http://sites.google.com/site/amtworkshop2010/
Submission Deadline: March 3, 2010, 11:59PM (note deadline extension)
Creating Speech and Text Language Data With Amazon's Mechanical Turk
Amazon's Mechanical Turk is an online labor market that allows
you to pay people small sums of money to do "Human Intelligence Tasks",
or HITs. Tasks include labeling images, listening to short pieces of
audio, researching topics on the internet, and scrubbing database
records.
A number of recent papers have evaluated the effectiveness of using
Mechanical Turk to create annotated data for natural language
processing applications. The availability of Mechanical Turk for
creating annotated speech and text language data can fundamentally
change human language technology tasks. Open questions include:
- How can we ensure high quality annotations?
- What tools are available for obtaining complex annotations?
- What types of annotations and evaluations are possible when the cost
is dramatically reduced?
This work will explore uses of Mechanical Turk in several ways:
- Shared task (application deadline passed): What can you do with $100
and Mechanical Turk? Participants will be
given a budget to spend on Mechanical Turk and submit papers describing the
results of their experience.
- General papers: These papers will explore general issues with using
Mechanical Turk for language processing research.
Important Dates
Mar 3, 2010, 11:59pm Paper submission deadline
Mar 30, 2010 Notification of acceptance
Apr 12, 2010 Camera-ready papers due
Jun 5-6, 2010 Workshop at NAACL in Los Angeles, CA (exact date TBA)
Workshop Website
http://sites.google.com/site/amtworkshop2010/
Organizing Committee
Chris Callison-Burch (Johns Hopkins University)
Mark Dredze (Johns Hopkins University)
Program Committee
Jordan Boyd-Graber (UMD)
Michael Bloodgood (HLTCOE)
Bob Carpenter (Alias-i)
James Dennis (J2 Labs)
Maxine Eskenazi (CMU)
Nikesh Garera (Kosmix)
Jim Glass (MIT)
Alex Gruenstein (Google)
Janna Hamaker (Amazon)
Benjamin Lambert (CMU)
Alexandre Klementiev (JHU)
Ian McGraw (MIT)
Scott Novotney (JHU)
Brendan O'Connor (CMU)
Massimo Poesio (University of Essex)
Joseph Reisinger (UT Austin)
Ted Sandler (Amazon)
Stephanie Seneff (MIT)
Rion Snow (Stanford / Twitter)
We thank Amazon Mechanical Turk and CrowdFlower for supporting the shared task.