Call for Participation: Scaling Up Multilingual Evaluation Shared Task on Performance Prediction

[Vishrav Chaudhary]

(Apologies for cross-posting)

Call for Participation: Scaling Up Multilingual Evaluation Shared Task on Performance Prediction

We are excited to announce the first edition of the Shared Task on Performance Prediction at SUMEval 2022.  The 1st Workshop on Scaling Up Multilingual Evaluation (SUMEval 2022) will be a full-day event taking place on November 24, 2022 (in a hybrid mode) and will be co-located with AACL-IJCNLP 2022 in Taipei, Taiwan.

Shared Task URL: https://www.microsoft.com/en-us/research/event/sumeval-2022/shared-task/

***Overview***

The task of performance prediction is to be able to accurately predict the performance of a model on a set of target languages. These languages may be present in the fine-tuning data (few-shot training) or may not be present (zero-shot training). The languages used for fine-tuning are referred to as pivots, while the languages that we would like to evaluate model on are targets. This shared task will consist of building a machine learning model that can accurately predict the performance of a multilingual model on languages and tasks that we do not have test data for, given accuracies of models on various combinations of pivot and target pairs.

***Challenge details***

We are releasing a dataset containing evaluation scores of multiple MMLMs on different tasks and languages. These scores can be used to train models that can predict how MMLMs trained on different pivot configurations will perform on target languages. The task is now to predict the model’s performance, given the training configuration and test languages.

Predictions will need to be made on test languages included in the training data, as well as surprise languages. For more details on the task formulation, please refer to the papers at the bottom of this page.


***Evaluation procedure and Baseline numbers***

Evaluation will be done in three conditions: LOLO (Leave One Language Out), LOCO (Leave One Configuration Out) and Surprise Languages. Please check the Readme file in the folder for more details.

***Challenge Timeline***

July 1 2022: Baseline numbers release
Last week of July 2022: Test set release and leaderboard opens
July 31 2022: Challenge ends
August 25 2022: Paper submission deadline

For questions and comments regarding the workshop please contact the organizers at sum...@microsoft.com

Please also follow the SUMEVal twitter account (@sum_eval) for regular updates!

Looking forward to your submissions,

Best,
On behalf of the SUMEval 2022 Organizers,

--

Vishrav Chaudhary
Senior Principal Researcher 

Microsoft Turing

Bellevue, WA

[Vishrav Chaudhary]

(Apologies for cross-posting)

Call for Participation: Scaling Up Multilingual Evaluation Shared Task on Performance Prediction

We are excited to announce that the evaluation server for non surprise languages is now open and the test sets have been released. 

We will be releasing the test sets for the surprise languages in a few days.

Submission instructions: https://github.com/microsoft/Litmus/tree/main/SumEval

Shared Task URL: https://www.microsoft.com/en-us/research/event/sumeval-2022/shared-task/

For questions and comments regarding the workshop please contact the organizers at sum...@microsoft.com
Please also follow the SUMEVal twitter account (@sum_eval) for regular updates!

Looking forward to your submissions,

Best,
On behalf of the SUMEval 2022 Organizers,
--

Vishrav Chaudhary
Senior Principal Researcher 

Microsoft Turing

Bellevue, WA

[Vishrav Chaudhary]

(Apologies for cross-posting)

Call for Participation: Scaling Up Multilingual Evaluation Shared Task on Performance Prediction

We are excited to announce that the evaluation server for both non surprise and surprise languages is now open and all the test sets have been released. 

Submission instructions: https://github.com/microsoft/Litmus/tree/main/SumEval

Shared Task URL: https://www.microsoft.com/en-us/research/event/sumeval-2022/shared-task/

August 15 2022: Challenge ends

August 25 2022: Paper submission deadline

For questions and comments regarding the workshop please contact the organizers at sum...@microsoft.com
Please also follow the SUMEVal twitter account (@sum_eval) for regular updates!

Looking forward to your submissions,

Best,
On behalf of the SUMEval 2022 Organizers,
--

Vishrav Chaudhary
Senior Principal Researcher 

Microsoft Turing

Bellevue, WA