Non-English summarization

Skip to first unread message


Apr 28, 2021, 11:54:48 AMApr 28
to gem-benchmark
Hi folks,

Thank you for creating the wonderful GEM benchmark!

The shared task contains the MLSum dataset used for evaluating multi-lingual summarization. However, our NLG model is applicable to English text only. Would it be possible for us to only submit English results on MLSum to the shared task, ignoring non-English ones?


Best regards,

Simon Mille

Apr 28, 2021, 12:23:34 PMApr 28
to bin, gem-benchmark
Hi Bin,

as far as I know you can submit for whichever dataset you want; note however that, in your case, you'd be asked to generate outputs for all the English MLSum test sets (official test set and special test sets) with your model. Here's some text from the official website:

While we highly encourage participation in the shared task even for a single dataset, we ask you to please submit outputs for all possible challenge sets to help us assess your submission.

hope this helps!

You received this message because you are subscribed to the Google Groups "gem-benchmark" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
To view this discussion on the web visit
For more options, visit
Reply all
Reply to author
0 new messages