[Metrics Task] Test sets now available!

[Nitika Mathur]

Dear all,

 

The metrics task inputs are now available: see the metrics task webpage for the download link and instructions on how to submit your metric. 

We are looking forward to your submissions by August 2nd!

Nitika,
on behalf of the Metrics Task organizers

[Mateusz Krubinski]

Dear all,

I think one of the system outputs has an incorrect number of lines:

     177 ./system-outputs/challengeset/en-de/challengeset.en-de.hyp.system2.de
     140 ./system-outputs/challengeset/en-de/challengeset.en-de.hyp.system1.de
     140 ./references/challengeset.en-de.ref.ref-A.de
     140 ./sources/challengeset.en-de.src.en

how should we approach it?

Best,

Mateusz Krubiński

--
You received this message because you are subscribed to the Google Groups "Workshop on Statistical Machine Translation" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wmt-tasks+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wmt-tasks/CAM87Ymdf7EPzKVwuEmmp3UKzi8yoOGSPn2DKtHS8ZVrnyeQvdg%40mail.gmail.com.

[Nitika Mathur]

Hi Mateusz,

Thanks a lot for flagging this. We'll look into it and get back to you ASAP. Please let us know if you have any other questions.

Nitika

To view this discussion on the web visit https://groups.google.com/d/msgid/wmt-tasks/CANs10qU1u8CFyLxP8fcHMHzNg4UkdPJBka8ZF5QtNxuBqpsyhA%40mail.gmail.com.

[Nitika Mathur]

Dear all,

We've updated the source, reference and system outputs for the en-de challenge set.  They should all have 177 lines. Thanks to Mateeusz for bringing this to our attention.

Best,

Nitika

[Markus Freitag]

Hello everyone,

We had to make a couple of adjustments to the WMT21 metric task test sets. Please re-download the test sets that contains significant changes.

Due to that, we will postpone the submission deadline for the WMT 21 metric task to August 9th.

Please be aware that you can submit a paper describing your metric to the WMT conference, but the deadline (08/05) for that is before the actual metric submission deadline (08/09). You can update your results later in the final version of your paper.

Thanks and sorry for the inconvenience,

Markus,

on behalf of the Metrics Task organizers

[Aaron Poet]

Hi Markus and Metric task organizers,

We are in process of writing up our metric description paper, however, it is somehow short of time regarding the deadline of 5th August. So we wonder if we can submit the metric description paper several days after. since the official deadline of metric system result submission is extended to 9th already?

Kind regards,

Lifeng

To view this discussion on the web visit https://groups.google.com/d/msgid/wmt-tasks/78CEF7B8-8278-41A9-8BAD-123E14DA2BD7%40gmx.de.

--

PhD Candidate, Msc. (Software Engineering, excellent), Bsc. (Math, GPA80/100)

Google-Scholar , Presentation-slides,  Research-Gate

Google-site Linkedin,

ADAPT Research Centre 

School of Computing, DCU

[Philipp Koehn]

Hi,

to not cause too much friction in the submission review process, it would be good to submit a draft paper 

by the deadline and then work on a polished version for the camera-ready deadline. System description

papers go through a lenient review process, so the paper would be accepted even as a draft.

-phi

To view this discussion on the web visit https://groups.google.com/d/msgid/wmt-tasks/CAETj864Ls4kCmzo0FWQMOUC0JWPXzrZUVwbmThJzdNc%3D7EHWjw%40mail.gmail.com.

[Aaron Poet]

Thanks Philipp for the response. 

To view this discussion on the web visit https://groups.google.com/d/msgid/wmt-tasks/CAAFADDB2V9YGNK%2Bb3E8%3Dv6gkvYv1xgT%3DtkweTAY%2B5eEsxn7%3D6g%40mail.gmail.com.

[Markus Freitag]

Hi,

Thanks for answering the question, Philip.

Yes, please submit a paper describing your metric to Wmt. The final version can be updated with new numbers and insights that you get in between.

Thanks and sorry for the strange schedule :-)

Markus

To view this discussion on the web visit https://groups.google.com/d/msgid/wmt-tasks/CAETj865MDe4Oc%3DpiN1qP%2BWpa5J_B2nHYMFRgDh_GsmQNj0BhtQ%40mail.gmail.com.

[Markus Freitag]

Hi everyone,

This is a friendly reminder that we ask all participants of the WMT 2021 metric task to enter your submission to the following spreadsheet:

https://docs.google.com/spreadsheets/d/18BZyGmfxoYK6nq4G6cK_JQO-9BuZ_Ik2SnDiSST7BVU/edit?usp=sharing

Thanks!

Markus on behalf of the metric team