Sign language task: updates and second call

[Mathias Müller]

Dear colleagues

(Apologies for cross-posting)

Below is an abridged version of the second call for the sign translation task.

Best regards

Mathias

-----------------------------------------------

Second Call for Participation: WMT Shared Task on Sign Language Translation (WMT-SLT)

We are excited to announce the first edition of the Shared Task on Sign Language Translation (WMT-SLT) at WMT 2022, co-located with EMNLP 2022.

*SECOND CALL IN INTERNATIONAL SIGN*

https://youtu.be/9MWLsVXIGVg

*NEW DEVELOPMENTS SINCE THE FIRST CALL*

(1) Release of training data and development data

Please see https://www.wmt-slt.com/data for more information.

(2) Release of baseline code

We now provide code to train baseline systems in a public Github repository. The codebase contains scripts to preprocess data, train, translate and evaluate models.

Please see https://www.wmt-slt.com/tools for more information.

*OVERVIEW*

This shared task is concerned with automatic translation between signed and spoken languages. The task is novel in the sense that it requires processing visual information (such as video frames or human pose estimation) beyond the well-known paradigm of text-to-text machine translation.

We provide training data for one language pair: Swiss German Sign Language (DSGS) and German (DE).

Translation from DSGS to German is our primary translation direction: submitted systems are ranked on a leaderboard and we will provide baseline systems. Systems translating from German to DSGS are not ranked on the leaderboard while the task is running, but we still encourage participants to submit such systems.

We aim to provide human evaluation for all submitted systems, regardless of the translation direction.

We will ask participants to submit a system description paper to WMT 2022.

*IMPORTANT DATES*
* 12 June 2022: Release of training data
* 12 June 2022: Release of baseline systems
* 14 July 2022: Release of test data
* 28 July 2022: System submission deadline
* 4 August 2022: System paper abstract submission deadline
* 7 September 2022: Paper submission deadline
* 9 October 2022: Paper acceptance notification
* 16 October 2022: Camera-ready version due
* 7-8 December 2022: Overview of results at WMT
All deadlines are AoE (Anywhere on Earth). Dates are in line with general deadlines of WMT.

*MORE INFORMATION*
For more information, please visit our website: www.wmt-slt.com
If you have any comments and/or questions, do not hesitate to ask a question on the mailing list: wmt-...@googlegroups.com or contact us: mmue...@cl.uzh.ch.

Mathias

[ZongYao Li]

Hello

 Can you please provide the devbleu value of the baseline generated by using the standard baseline model and openpose data?
Thanks

[Mathias Müller]

Hi

Thanks for your email! This is a great suggestion, I have added a new table here:

https://github.com/bricksdont/sign-sockeye-baselines#baseline-scores-examples

There is no "standard baseline model", so I included 3 different ones.

Automatic metric scores are extremely low for this task. This means that while for other MT tasks, a BLEU score of 1.0 indicates that something went wrong in your pipeline, for our task that is not necessarily the case.

Please let me know if this helps and best regards

Mathias

--
You received this message because you are subscribed to the Google Groups "Workshop on Statistical Machine Translation" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wmt-tasks+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wmt-tasks/0ec5631f-8432-43ba-a715-51cbd276cc01n%40googlegroups.com.

[ZongYao Li]

Mathias
Thank you for the reply. For a long time, I thought I was doing my experiments wrong.

Zongyao Li