Call for Papers : 1st Workshop on Scaling Up Multilingual Evaluation (SUMEval)
69 views
Skip to first unread message
Vishrav Chaudhary
Jul 10, 2022, 6:23:00 PM7/10/22
to wmt-...@googlegroups.com
(Apologies for cross-posting)
Massively Multilingual Language Models (MMLMs) are trained on around 100 languages of the world, however, most existing multilingual NLP benchmarks provide evaluation data in only a handful of these languages. The languages present in evaluation benchmarks are usually high-resource and largely belong to the Indo-European language family. This makes current multilingual evaluation unreliable and does not provide a full picture of the performance of MMLMs across the linguistic landscape. Although efforts are being made to create benchmarks that cover a larger variety of tasks, languages, and language families, it is unlikely that we will be able to build benchmarks covering all languages and tasks. Due to this, there is recent interest in alternate strategies for evaluating MMLMs, including performance prediction and Machine Translation of test data. We believe that this is an important yet relatively unexplored area of research that has the potential to make language technologies accessible to all.
*** Important Dates ***
Workshop Paper Submission Deadline: August 25, 2022
Notification of Acceptance: September 25, 2022
Camera-ready papers due: October 10, 2022
Workshop Date: November 23, 2022
All deadlines are 11:59 pm UTC -12h ("anywhere on Earth").
*** Papers ***
We invite submissions on topics that include, but are not limited to, the
following:
* Studies on scaling up multilingual evaluation
* Human evaluation of multilingual models
* Automated evaluation metrics for multilingual evaluation
* Studies on fairness and other aspects of evaluation
* Data sets, benchmarks or libraries for evaluating multi-lingual models
* Probing and analysis of multilingual models
For questions and comments regarding the workshop please contact the organizers at sum...@microsoft.com
Please also follow the SUMEVal twitter account (@sum_eval) for regular updates!
Looking forward to your submissions,
Best,
On behalf of the SUMEval 2022 Organizers,
Call for Papers in the 1st Workshop on Scaling Up Multilingual Evaluation (SUMEval)
The 1st Workshop on Scaling Up Multilingual Evaluation (SUMEval 2022) will be a full-day event taking place on November 24, 2022 (in a hybrid mode). This year’s SUMEval will be co-located with AACL-IJCNLP 2022 in Taipei, Taiwan.
Workshop URL: https://aka.ms/sumeval
Workshop URL: https://aka.ms/sumeval
Massively Multilingual Language Models (MMLMs) are trained on around 100 languages of the world, however, most existing multilingual NLP benchmarks provide evaluation data in only a handful of these languages. The languages present in evaluation benchmarks are usually high-resource and largely belong to the Indo-European language family. This makes current multilingual evaluation unreliable and does not provide a full picture of the performance of MMLMs across the linguistic landscape. Although efforts are being made to create benchmarks that cover a larger variety of tasks, languages, and language families, it is unlikely that we will be able to build benchmarks covering all languages and tasks. Due to this, there is recent interest in alternate strategies for evaluating MMLMs, including performance prediction and Machine Translation of test data. We believe that this is an important yet relatively unexplored area of research that has the potential to make language technologies accessible to all.
*** Important Dates ***
Workshop Paper Submission Deadline: August 25, 2022
Notification of Acceptance: September 25, 2022
Camera-ready papers due: October 10, 2022
Workshop Date: November 23, 2022
All deadlines are 11:59 pm UTC -12h ("anywhere on Earth").
*** Papers ***
We invite submissions on topics that include, but are not limited to, the
following:
* Studies on scaling up multilingual evaluation
* Human evaluation of multilingual models
* Automated evaluation metrics for multilingual evaluation
* Studies on fairness and other aspects of evaluation
* Data sets, benchmarks or libraries for evaluating multi-lingual models
* Probing and analysis of multilingual models
For questions and comments regarding the workshop please contact the organizers at sum...@microsoft.com
Please also follow the SUMEVal twitter account (@sum_eval) for regular updates!
Looking forward to your submissions,
Best,
On behalf of the SUMEval 2022 Organizers,
--
Vishrav Chaudhary
Senior Principal Researcher
Senior Principal Researcher
Microsoft Turing
Bellevue, WA
Vishrav Chaudhary
Aug 26, 2022, 1:11:09 PM8/26/22
to wmt-...@googlegroups.com
(Apologies for cross-posting)
**Deadline Extension**
The workshop paper submission deadline has now been extended to September 1 2022.
Please refer to the following link for paper submission instructions: https://www.microsoft.com/en-us/research/event/sumeval-2022/paper-submission/
For questions and comments regarding the workshop please contact the organizers at sum...@microsoft.com
Please also follow the SUMEVal twitter account (@sum_eval) for regular updates!
Looking forward to your submissions,
Best,
On behalf of the SUMEval 2022 Organizers
Please also follow the SUMEVal twitter account (@sum_eval) for regular updates!
Looking forward to your submissions,
Best,
On behalf of the SUMEval 2022 Organizers
Vishrav Chaudhary
Sep 9, 2022, 2:09:15 AM9/9/22
to wmt-...@googlegroups.com
(Apologies for cross-posting)
**Deadline Extension**
The workshop paper submission deadline has now been extended to September 23, 2022.
In addition to regular papers submitted to the workshop, we will also accept papers that were submitted elsewhere and papers with AACL and ARR reviews.Please refer to the following link for paper submission instructions: https://www.microsoft.com/en-us/research/event/sumeval-2022/paper-submission/
For questions and comments regarding the workshop please contact the organizers at sum...@microsoft.com
Please also follow the SUMEVal twitter account (@sum_eval) for regular updates!
Looking forward to your submissions,
Best,
On behalf of the SUMEval 2022 Organizers
Please also follow the SUMEVal twitter account (@sum_eval) for regular updates!
Looking forward to your submissions,
Best,
On behalf of the SUMEval 2022 Organizers
Reply all
Reply to author
Forward
0 new messages