WMT21 Metrics Task update

[Nitika Mathur]

Dear all,

We need to postpone the date of release of MT system outputs as the data is not ready yet. We will also update the deadline of metric score submissions. Apologies for the delay, and here are the new dates:

System outputs ready to download	July 26, 2021
Submission deadline for metrics task	August 02, 2021(AoE)

We have more information on evaluation of metrics: At the system level, we will rank metrics based on their Pearson Correlation with human scores. At the segment level, we will use Kendall Tau's correlation Also, we've decided not to run document level evaluation this year.

If you intend to participate, please fill out this form so we can keep track of submissions https://docs.google.com/spreadsheets/d/18BZyGmfxoYK6nq4G6cK_JQO-9BuZ_Ik2SnDiSST7BVU/edit?usp=sharing

Please let us know if you have any questions.

Best,

Metrics Task team 2021

[Mateusz Krubinski]

Dear Metrics Task team,

thank you for the update.

Just to be sure - the "human scores" used to rank the metrics will be the MQM or the DA scores?

Thank you and best regards,

Mateusz Krubiński

--
You received this message because you are subscribed to the Google Groups "Workshop on Statistical Machine Translation" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wmt-tasks+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wmt-tasks/CAM87YmcqPcWTq6pZKoNx3Z2sMjP9Cn1YprLpNKo%2BDCsJzyy_iA%40mail.gmail.com.

[Nitika Mathur]

Hi Mateusz,

We will be using MQM scores on the en-de, en-ru and zh-en language pairs  for our primary evaluation, and will use these to draw all main conclusions.

Note that we will also report results on DA scores for all language pairs.

Best,

Nitika

To view this discussion on the web visit https://groups.google.com/d/msgid/wmt-tasks/CANs10qXr6Mjfa-6cAmRZmPmi0vTH2rbEfacQNcy05rykvo411A%40mail.gmail.com.

[Eleftherios Avramidis]

Hi,

is there going to be a similar delay for the Test Suite track?  We haven't received any outputs yet and they were last supposed to come by the 17th of July.

If for any reason we cannot have the full outputs soon, we would appreciate if we could have a partial output, even from some of the systems, particularly for the en-de. Our test suite is semi-automated, and when new language pairs are included, we need substantial human effort to right new rules. We were planning to do that manual effort on this year's SoTA systems between the original data release date (30 of June) and the paper submission deadline (5 of August). Unfortunately given those delays this effort becomes highly impossible because there is only a bit more than a working week left. So yes, a partial release would help things a bit.

best
Lefteris

--
You received this message because you are subscribed to the Google Groups "Workshop on Statistical Machine Translation" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wmt-tasks+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wmt-tasks/CAM87YmcqPcWTq6pZKoNx3Z2sMjP9Cn1YprLpNKo%2BDCsJzyy_iA%40mail.gmail.com.

-- 
-------------------------------------------------------------
Eleftherios Avramidis
Senior Researcher; 
Speech and Language Technology, Interactive Textiles
Alt Moabit 91c, 10559 Berlin

Deutsches Forschungszentrum für Künstliche Intelligenz GmbH
Trippstadter Strasse 122, 67663 Kaiserslautern, Germany 

Geschäftsführung:
Prof. Dr. Antonio Krüger (Vorsitzender)
Helmut Ditzer

Vorsitzender des Aufsichtsrats: 
Dr. Gabriël Clemens
Amtsgericht Kaiserslautern, HRB 2313
-------------------------------------------------------------

[Ondrej Bojar]

Dear Lefteris, all,

the delay falls primarily on my head. I am finally out of the most critical other burden and I should send the testsuites in a day or two.

I am really sorry that it's me this year who is causing the stress...

Apologies,
Ondrej.

>> <mailto:wmt-tasks+...@googlegroups.com>.

>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/wmt-tasks/CAM87YmcqPcWTq6pZKoNx3Z2sMjP9Cn1YprLpNKo%2BDCsJzyy_iA%40mail.gmail.com

>> <https://groups.google.com/d/msgid/wmt-tasks/CAM87YmcqPcWTq6pZKoNx3Z2sMjP9Cn1YprLpNKo%2BDCsJzyy_iA%40mail.gmail.com?utm_medium=email&utm_source=footer>.

>
> --
> -------------------------------------------------------------
> Eleftherios Avramidis
> Senior Researcher;
> Speech and Language Technology, Interactive Textiles
> Alt Moabit 91c, 10559 Berlin
>
> Deutsches Forschungszentrum für Künstliche Intelligenz GmbH
> Trippstadter Strasse 122, 67663 Kaiserslautern, Germany
>
> Geschäftsführung:
> Prof. Dr. Antonio Krüger (Vorsitzender)
> Helmut Ditzer
>
> Vorsitzender des Aufsichtsrats:
> Dr. Gabriël Clemens
> Amtsgericht Kaiserslautern, HRB 2313
> -------------------------------------------------------------
>

> --
> You received this message because you are subscribed to the Google Groups
> "Workshop on Statistical Machine Translation" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to wmt-tasks+...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/wmt-tasks/e77384d5-af7a-c0d5-e2cc-703c4fea7a41%40dfki.de.

--
Ondrej Bojar (mailto:o...@cuni.cz / bo...@ufal.mff.cuni.cz)
http://www.cuni.cz/~obo