[Metrics Task] Test sets now available

[Nitika Mathur]

Happy to announce that the metrics task inputs are now available! We have 11 language-pairs available in the generaltest2024 testset, as well as 5 additional challenge sets including african languages; for general purpose metrics, we expect participation in all language-pairs. 

Submissions are now open via Codabench

Deadline for submissions has been extended by 12 hours, and is now 12:00 pm July 31 (AoE) or 00:00 August 1st (UTC) 

Process:

1. Register your metric here, if you haven’t already
2. Create an account on Codabench.
   • You’re allowed one primary submission for a reference-based metric, and one primary submission for a reference-free metric. If you are submitting two metrics that have widely different approaches, for example, one LLM-based metric and one lexical metric, then create 2 accounts on Codabench.
3. Download the data (link; link also available on Codabench)
4. Prepare your scores and submit your scores on Codabench following the instructions at the "Getting Started -> evaluation" page at Codebench

Please contact us if you see any issue or have any other questions.

Kind regards,

Metrics Task organizers wmt-m...@googlegroups.com