WMT09 Manual Evaluation
Josh Schroeder
Thanks again for participating in the ACL WMT 2009 Shared Tasks. We're
ready to start the manual evaluation of the translations. To begin,
please create a user account for yourself by going to
http://www.statmt.org/wmt09/judge/
You'll be asked to indiciate your site so we can optionally filter out
self-judgments. After registering you can then enter your username
and password and start the evaluation process.
When you log in you will see a list of all the shared translation
tasks. The tasks names tell you the language pair and what type of
judgment you'll be doing. Please only evaluate translations into a
language that you speak fluently. You should work through several
tasks, and can also do the same task multiple times. Feel free to
evaluate tasks that you didn't participate in.
We ask that each site contribute about 8 hours to this effort, which
can be distributed over multiple annotators.
When you select a task, you'll be asked for two different types of
judgments:
(1) Ranking the translations of whole sentences
(2) Editing the output a machine translation system
The second type of annotation is new this year. In the second week of
the evaluation period we will be asking people to judge the edited
sentences to see if they have the same meaning as the reference. The
goal is to try to test the understandability of each system.
Finally, the deadline for completing your judgments is FRIDAY JANUARY
30.
We think that this evaluation campaign will be a success, and we're
looking forward to seeing you in Athens.
Chris Callison-Burch (Johns Hopkins University)
Philipp Koehn (University of Edinburgh)
Christof Monz (University of London)
Josh Schroeder (University of Edinburgh)
--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.
Josh Schroeder
By popular demand, we've added editing of output for all languages,
not just English. If you're fluent in Hungarian, Czech, French, German
or Spanish, please consider editing a few sentences.
Thanks,
Josh
Ondrej Bojar
a suggestion for the editing task next year: while the source text is hidden on
purpose, the target (hypothesis) side context should be made available. I often
have the feeling I would be able to correct the sentence, if only I could guess
the type of source domain or claim made.
Ondrej.
--
Ondrej Bojar (mailto:o...@cuni.cz / bo...@ufal.mff.cuni.cz)
http://www.cuni.cz/~obo