Call for Participation: "Help us break LLMs" - Test suite sub-task of the Ninth Conference on Machine Translation (WMT24)
Eleftherios Avramidis
The “test suites” sub-task will be included for the sixth time in
the General MT Shared Task of the Conference on Machine
Translation (WMT24).
*OVERVIEW*
Test suites are custom extensions to the test sets of the General
MT Shared Task, constructed so that they can focus on concrete
aspects of the MT output. They consist of a source-side test-set
and a customized evaluation service. As opposed to the standard
evaluation process which produces generic quality scores, test
suites often produce separate fine-grained results for each
phenomenon.
Since the usage of LLMs for translation is getting more popular,
and we are expecting more LLMs participations in WMT this year,
the theme of this year’s test suite sub-task is "Help us break
LLMs", i.e. to reveal weaknesses and serious flaws of LLMs when
translating, hidden within the overall high-quality generation.
*IMPORTANT DATES*
- 11th April: Test suite source texts may be submitted for a pre-run on SoTA MT systems
- 12th June: Test suite source texts must reach us
- 11th July: Translated test suites shipped back to test suites authors:
- TBC - August: Test suite description and analysis paper
- 12th-13th November: Conference
Potential participants are kindly requested to fill in this form
https://forms.office.com/e/e4JuMTSWFF
Further information can be found in the dedicated page of the WMT
website
http://www2.statmt.org/wmt24/testsuite-subtask.html
-- Eleftherios Avramidis, senior researcher German Research Center for Artificial Intelligence (DFKI) departments: Design Research eXplorations, Speech and Language Technology short name: Lefteris, (pronouns: he/him), languages: English, German, Greek Website: https://www.dfki.de/~elav01 Address: Alt Moabit 91c, 10559 Berlin, Germany Tel.: +49 30 23895 1806 Sec.: +49 30 23895 1800 Fax.: +49 30 23895 1810 Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern, Germany Geschäftsführung: Prof. Dr. Antonio Krüger, Helmut Ditzer Vorsitzender des Aufsichtsrats: Dr. Ferri Abolhassan Amtsgericht Kaiserslautern, HRB 2313
nicolas...@gmail.com
--
You received this message because you are subscribed to the Google Groups "WMT: Workshop on Machine Translation" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wmt-tasks+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wmt-tasks/48c3892e-4fea-4af7-bc62-3a7aedd21675%40dfki.de.