Re: ROUGE Evaluation Metric

Skip to first unread message

Ashok Urlana

Sep 8, 2022, 12:59:33 AM9/8/22
Dear Organizers,

Please help with the following questions
1. Which metric is used to compute the ROUGE score for Hindi and Gujarati? If possible, release the evaluation script.
2. Are we allowed to use different models in the testing phase that are not used in the validation phase?


Parth Mehta

Sep 8, 2022, 1:14:04 AM9/8/22
to Ashok Urlana, ILSUM
Dear Ashok,

Same rouge metrics will be used for evaluation of all languages, as ROUGE is language independent. The only language dependent component is a stemmer which we will not be using.

We will release evaluation script after test submission closes, meanwhile we encourage all teams to evaluate using the submission platform and leaderboard.

And yes, you are free to use any new models during test phase irrespective of what you submitted in validation phase. However, the maximum submissions in test phase will be limited to 3, so we suggest choosing your models wisely. 

Let us know if you have any other questions.


You received this message because you are subscribed to the Google Groups "ILSUM" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
To view this discussion on the web visit
For more options, visit
Reply all
Reply to author
0 new messages