WMT22 - News task is going towards multidomain

225 views
Skip to first unread message

Tom Kocmi

unread,
Feb 4, 2022, 11:38:09 AM2/4/22
to wmt-tasks

Dear All,

 

I am happy to announce plans for the WMT22 news translation shared task. We are preparing to incorporate significant changes for this year and we want to share our plans with the community and collect feedback.

 

Here is a list of the main changes:

 

  • Multi-domain direction: we are taking a new direction with the News task in form of testing the general capabilities of MT systems. Thus as an approximation, blind tests for all languages will contain (up to) four different domains. Likely news, social, conversational, and e-commerce.
  • Aggregated blind testsets across tasks: we are cooperating with other tasks to combine blind testsets, thus participants from each shared task will translate blind testsets from other shared tasks if given language pair is shared. This will allow comparisons of participants across tasks (for example how news task participants compete against biomedical system submissions). Each shared task will evaluate their blind testsets separately. Blind testsets from different tasks won’t be used for human evaluation and ranking of systems at News task translation.
  • Rethinking human evaluation: TBA
  • Removing the requirement expecting participants to annotate 10 hours per primary system. Instead, participants of the news shared task will be required to submit a system description paper if they submit the primary system.
  • Minor change: we allow pretrained models in constrained setup (only those that were publicly available before February 2022).
  • With this fresh direction, we are changing the name from “News translation task” to “General MT task”

 

We welcome any comments and suggestions from the community regarding our plans and will consider all such feedback. As there could be a lot of questions and detailed discussion, you may use this document for giving feedback and reading our replies:

https://1drv.ms/w/s!Aq0goPMF_LnlhPYwO46qJGJFcq51ig?e=j6ofH4

 

Confirmed list of languages at WMT22 General MT task: Chinese-EN, Czech-EN, German-EN, German-French, Japanese-EN, Russian-EN, and several low-resource languages (TBA). The deadline for system submissions is planned for June (webpage https://www.statmt.org/wmt22/ will be available during the next week).

 

Thank you and we are looking forward to your system submissions,

On behalf of General MT task organizers,

Tom Kocmi

 

Tom Kocmi

unread,
Feb 4, 2022, 1:20:06 PM2/4/22
to wmt-tasks
Hi All,

I apologize for the broken link. Here is a new link (we will place it on the webpage as well).

Have a lovely weekend,
Tom 
(Germany, he/him)


> Dear All,
>
>
>
> I am happy to announce plans for the WMT22 news translation shared task. We
> are preparing to incorporate significant changes for this year and we want
> to share our plans with the community and collect feedback.
>
>
>
> Here is a list of the main changes:
>
>
>
>   - Multi-domain direction: we are taking a new direction with the News

>   task in form of testing the general capabilities of MT systems. Thus as an
>   approximation, blind tests for all languages will contain (up to) four
>   different domains. Likely news, social, conversational, and e-commerce.
>   - Aggregated blind testsets across tasks: we are cooperating with other

>   tasks to combine blind testsets, thus participants from each shared task
>   will translate blind testsets from other shared tasks if given language
>   pair is shared. This will allow comparisons of participants across tasks
>   (for example how news task participants compete against biomedical system
>   submissions). Each shared task will evaluate their blind testsets
>   separately. Blind testsets from different tasks won’t be used for human
>   evaluation and ranking of systems at News task translation.
>   - Rethinking human evaluation: TBA
>   - Removing the requirement expecting participants to annotate 10 hours

>   per primary system. Instead, participants of the news shared task will be
>   required to submit a system description paper if they submit the primary
>   system.
>   - Minor change: we allow pretrained models in constrained setup (only

>   those that were publicly available before February 2022).
>   - With this fresh direction, we are changing the name from “News

>   translation task” to “General MT task”
>
>
>
> We welcome any comments and suggestions from the community regarding our
> plans and will consider all such feedback. As there could be a lot of
> questions and detailed discussion, you may use this document for giving
> feedback and reading our replies:
>

Andreas Eisele

unread,
Mar 22, 2022, 6:19:02 AM3/22/22
to Workshop on Statistical Machine Translation
Dear all,

I had a look at this year's MT shared task (https://www.statmt.org/wmt22/translation-task.html), and I was a bit surprised about one detail:
The language pair sah-ru is characterized as low resource, but closely-related. 
According to my knowledge (and Wikipedia), these languages aren't even from the same language family, so while there might be considerable lexical overlap, I don't think calling them closely related is appropriate.
Or am I missing something here?

Best regards,
Andreas

Adam Dobrowolski

unread,
Apr 8, 2022, 8:43:39 AM4/8/22
to Workshop on Statistical Machine Translation
Hello,

Can you write a bit more about the domains that would be used as the multi-domain testset? "News" and "coversational" is understandable. But what it means "social"? Is it about social media texts (FB/Twitter)? What is e-commerce? Texts on e-commerce pages like e-bay: sales, promotions etc?

We are provided with ParaCrawl v9. But for EN-RU the version v9 contains only 5M sentences while the previous v8 version contained more than 10M. Can we use v8 version for constrained submissions?
Since general MT submissions would be checked against other shared tasks can we use other tasks' (biomedical?) resources in constrained path?

Will there be available any monolingual corpus for Ukrainian?

Regards,
Adam Dobrowolski
Samsung R&D Poland

Tom Kocmi

unread,
Apr 8, 2022, 9:04:24 AM4/8/22
to wmt-...@googlegroups.com

Hi Adam,

 

those are great questions.

 

> domains

We do not plan to disclose any information about the domains, we only provided rough estimate what could be expected as this is a first year. The goal of General MT shared task is to investigate general capabilities of MT systems (which itself is open question, how to define it). We decided to simplify it by selecting few domains which we won’t be evaluating individually. Moreover, domains can slightly differ across languages as it is difficult to obtain monolingual resources (we are targeting data created in 2021 and 2022 whenever possible).

 

Regarding ParaCrawl and biomedical, at this moment only what is on the webpage is allowed for constrained task. Although we still have discussions about extending the constrained task and allowing larger quantities of training data, so potentially we will extend the training set.

 

As for monolingual data for Ukrainian, if you know about any publicly available Ukrainian data (both mono and parallel) that are not allowed at this moment for constrained task, send me a message and we can add them (if reasonable quality and quantity). Martin Popel potentially knows about other efforts for Ukrainian.

 

Have a lovely day,

Tom

 

 

From: wmt-...@googlegroups.com <wmt-...@googlegroups.com> On Behalf Of Adam Dobrowolski
Sent: Friday, April 8, 2022 2:44 PM
To: Workshop on Statistical Machine Translation <wmt-...@googlegroups.com>
Subject: [EXTERNAL] Re: WMT22 - News task is going towards multidomain

 

Some people who received this message don't often get email from adobrow...@gmail.com. Learn why this is important

--
You received this message because you are subscribed to the Google Groups "Workshop on Statistical Machine Translation" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wmt-tasks+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wmt-tasks/6378e013-f67a-4b3d-807e-4d19809867een%40googlegroups.com.

Reply all
Reply to author
Forward
0 new messages