judging edited machine translation output

[Chris Callison-Burch]

We have updated the WMT09 manual evaluation tasks. Now, instead of
editing the output of machine translation systems, we are asking you
to judge whether the edited translations are correct or not.

The new task shows the source and the reference along with some
context. You should supply simple Yes/No judgments about whether the
edited translations represent fully fluent and meaning-equivalent
alternatives to the reference.

The deadline for the manual evaluation is the end of Friday. If you
participated in the shared task, and your group has not yet completed
your 8 hour time commitment, then please log in and judge some
translations. The acceptability judgments are highest priority, but
the ranking statistics are still useful.
Here is the URL for the manual evaluation:
http://statmt.org/wmt09/judge/

Thanks again for your participation!

Chris Callison-Burch
Philipp Koehn
Christof Monz
Josh Schroeder

[Chris Callison-Burch]

Hi Gregor,

Good point: if you know the source language, feel free to use that
instead of, or in addition to, the reference.

--Chris

On Jan 27, 2009, at 6:40 PM, in...@filmstudio.rwth-aachen.de wrote:

> Hi,
>
> in the "judge the edited" task -- should we go by the source or by
> the reference in the cases where the two say something completely
> different?
>
> For example, in the following sentence:
> "In the sky he is accompanied by the pride of his daughters who are
> also his fans on the earth, something not so common in America."
>
> the German translation
> "Im Himmel wird er stolz darauf sein, dass auf der Erde seine
> Töchter die gleiche Meinung wie seine Fans haben, was in Amerika
> nicht unbedingt üblich ist."
>
> would roughly translate as
> "In heaven, he will be proud that his daughters and his fans have
> the same opinion, which is not really usual in America."
>
> So, what should we do here?
>
> Gregor

> --
> Gregor Leusch
> leu...@i6.informatik.rwth-aachen.de
> Aachen University of Technology, Lab for Computer Science VI

[Yuval Marton]

Hi Chris,
Following the previous concern about the references, I have to say that reading some of the references makes me wonder if they were produced by humans (I hope I am not offending anyone here! :-)   Some references, I think, were not very fluent, and/or not very faithful to the source. For example :
In the production of metal processing a price increase of 13.2 percent, in the field of food, beverage, tobacco production 11 percent growth was seen.
(source: En el sector de la elaboración del metal se observó un aumento de un 13,2 por ciento, mientras que en la industria de alimentos, bebidas y tabacos, de peso determinante, hubo un aumento de 11 por ciento. )

or:
"I want to be a young mother, doesn't matter the candidate"
(source: "Quiero ser mamá joven; el candidato no me preocupa" )

So I decided to use my non-native (but hopefully good enough) English, and very poor Spanish, in order to judge translations as "Yes" if they were around the same quality as the reference (or better), and not vote them down if I thought the reference was not better.  Please feel free to ignore my judgments if you disagree with this policy.

-Yuval

[Philipp Koehn]

Hi,

if you can read the source, you should ignore the reference - but
you should not judge a faulty machine translation as good just because
the human translator did not do better.

-phi