Evaluation script

adam.ek

unread,

May 30, 2022, 5:14:23 AM5/30/22

to The 1st Shared Task on Multilingual Clause-level Morphology 2022

Dear organizers,

Will you provide the evaluation script that will be used to score the shared task before the final submission of our results? Can't seem to find it anywhere currently.

Best,
Adam

MRL 2022

unread,

May 30, 2022, 3:00:23 PM5/30/22

to adam.ek, The 1st Shared Task on Multilingual Clause-level Morphology 2022

Dear Adam,

Thank you very much for your inquiry. The shared task evaluation will use the edit distance metric for the generative, and precision based evaluation for the analysis tasks. We are in the process of finalizing our submission system, and we can make some example scripts available for aiding your evaluation.

Best wishes,

Organizers, 1st Shared Task on Multilingual Clause-level Morphology

--
You received this message because you are subscribed to the Google Groups "The 1st Shared Task on Multilingual Clause-level Morphology 2022" group.
To unsubscribe from this group and stop receiving emails from it, send an email to participants-mcmshare...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/participants-mcmsharedtask-2022/dbd5b7d8-705b-47cf-bddc-570f86fd615en%40googlegroups.com.

adam.ek

unread,

May 31, 2022, 4:53:21 AM5/31/22

to The 1st Shared Task on Multilingual Clause-level Morphology 2022

Ok, so for the evaluation task the precision is calculated for the grammatical features, but is accuracy calculated for the lemma also? Or is the edit distance used for the lemma? Since afaik, a system is supposed to generate both the "head" lemma and the features.

The evaluation script when you have it ready would be great, but no rush! Thanks for the quick response.