Dear all,
the “Test suites” sub-task will be included for the sixth time in
the General MT Shared Task of the Conference on Machine
Translation (WMT23).
*OVERVIEW*
Test suites are custom extensions to the test sets of the General MT Shared Task, constructed so that they can focus on particular aspects of the MT output. They cοnsist of a source-side test-set and a customized evaluation service. As opposed to the standard evaluation process which produces generic quality scores, test suites often produce separate fine-grained results for each phenomenon.
Given the massive improvement of MT and the emergence of LLMs
recently, test suites can be useful at revealing serious flaws of
otherwise highly scoring systems.
*IMPORTANT DATES*
Test suite source texts must reach us:
|
19th June |
Translated test suites shipped back to
test suites authors: |
27th July |
Test suite description and analysis
paper: |
TBC - September |
Potential participants are kindly requested to fill in this form
https://forms.gle/s4JEJt9WSqAzP74o6
*MORE INFORMATION*
The sub-task is co-ordinated by Ondrej Bojar and Eleftherios
Avramidis.
Further information can be found in the dedicated page of the WMT
website
http://www2.statmt.org/wmt23/testsuite-subtask.html
(apologies for multiple postings)
-- Dr. Eleftherios Avramidis DFKI GmbH, Alt-Moabit 91c, 10559 Berlin Tel. +49-30 238 95-1806 Fax. +49-30 238 95-1810 ------------------------------------------------------------- Deutsches Forschungszentrum für Künstliche Intelligenz GmbH Trippstadter Strasse 122, D-67663 Kaiserslautern, Germany Amtsgericht Kaiserslautern, HRB 2313 -------------------------------------------------------------