WMT24 Metrics Task: Call for Participation

[Nitika Mathur]

Hi all,

The details of the WMT24 Metrics Task are up at 

http://www2.statmt.org/wmt24/metrics-task.html.  We are looking for both reference-based metrics and reference-free metrics to evaluate the quality of MT systems. We’ll be using expert-based MQM annotations on English-German, English-Spanish and Japanese->Chinese as the primary gold standard for evaluating metrics. 

We’ll be continuing the challenge sets subtask this year: we invite anyone to submit a new test suite and/or an analysis paper on metric behaviour for specific perturbations/phenomena (you’re welcome to resubmit previous years’ challenge sets, so we can analyse progress over the years!) 

New this year:

• New language pairs: English-Spanish and Japanese->Chinese 

• Additional 13 typologically diverse African languages in a challenge set

• We will be using the Codabench platform to improve the metric submission experience

Important dates:

Challenge sets submission deadline: 11th July

Metrics inputs ready to download: 23rd July

Metric submission deadline: 30th July

Metric scores for challenge sets distributed: 6th August

Paper submission deadline to WMT: 20th August

Please register your metric submissions here and challenge set submissions here so we can keep track of participants. 

Looking forward to your submissions,

Metrics 2024 team