[Chat translation task WMT20] updates on manual evaluations and reference translations

[Amin Farajian]

Dear Participants,

We would like to inform you that the results of the manual document-level direct assessments of all the primary submissions of the Chat Translation shared task at WMT 2020 are now available on the website http://www.statmt.org/wmt20/chat-task.html. 

Moreover, the reference translations of the test set are also available on the website.

Thank you very much,

Amin 

[LIANG DING]

Dear Dr. Amin Farajian,

Thank you for the effort to organize the manual evaluations of WMT Chat Translations~ 

I have a question, why is there only English-German in this manual evaluation result, but no opposite direction, i.e., German-English?

Thank you again for your efforts and look forward to your Reply

Liam

Amin Farajian <ma.fa...@gmail.com> 于2020年10月8日周四 下午6:40写道：

--
You received this message because you are subscribed to the Google Groups "Workshop on Statistical Machine Translation" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wmt-tasks+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wmt-tasks/8ED074F8-D847-4516-9473-52924C05645D%40gmail.com.

[Amin Farajian]

Dear Liam,

There were several factors that led us to perform the manual evaluation only on the agent side (English-German). The customer source text was originally in English and was translated into German. We realised that showing the noisy English text to the evaluators as the reference could be misleading for them impacting their evaluations. 

Moreover, in this setting it is usually fine for the agent to have a gist of the message; however there is extra responsibility in translating the agent since it has the company brand with it.

Therefore, we preferred to focus more on evaluating the agent translations more rigorously than to spend resources in evaluating the customer.

Hope this answers your question.

Best,

Amin

To view this discussion on the web visit https://groups.google.com/d/msgid/wmt-tasks/CAHZN6XBuPc8iq4Fh%3DG3PSQPAdQRxawYEC9aA%3DO1Uszxv0DSAvA%40mail.gmail.com.