per-language results

[nick arefyev]

Dear colleagues,

Thank you very much for publishing the leaderboard so soon after the end of the evaluation period, this is really helpful for us to start writing the participant papers. I feel it might be also important for our papers (and maybe for the task description paper as well) to see the best per-language results of each team. For instance, in our submission which was had best average ARI for 3 languages, the predictions for Russian were quit poor (ARI=0.05) but in another submission they are better (ARI=0.26). This is because from the competition rules we didn't realize the leaderboards will be based on the average ARI for 3 languages, so didn't aim to collect the best predictions in a single submission.

Probably, the simplest thing would be publishing a single (large) table containing the individual metrics for all submission from all teams in addition to the leaderboards. Then we could analyze this from different perspectives in our papers, e.g. seeing which methods are strong for a particular language.

Best wishes, Nikolay.

[Andrey Kutuzov]

Dear Nikolay,

We encourage all the participants to make their submissions public on
Codalab, if they want. Then, these submissions are visible to everyone
with all the scores for all the languages.

As for one large table with the scores for all the submissions, we will
probably create it at some point, but no guarantees it will happen soon
(Codalab does not support such an export, we'll have to parse their dumps).

Thus, the fastest way is again for all the teams to make their
best-per-language submissions public themselves. But of course we cannot
force anyone to do that.

> --
> You received this message because you are subscribed to the Google
> Groups "AXOLOTL-24" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to axolotl-24+...@googlegroups.com
> <mailto:axolotl-24+...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/axolotl-24/3b989587-18c6-4dd3-833f-b5d48bb14e7dn%40googlegroups.com <https://groups.google.com/d/msgid/axolotl-24/3b989587-18c6-4dd3-833f-b5d48bb14e7dn%40googlegroups.com?utm_medium=email&utm_source=footer>.

--
Andrey
Language Technology Group (LTG)
University of Oslo

[Mariia Fedorova]

Dear Nikolay,

We have discussed it and understood that the problem with one large table for all the submissions would be that we do not know which of them are meaningful (some may be test files just to make sure that codalab accepts the format etc., some may contain wrong method descriptions...). So the best and only possible solution for us organizers is letting participants decide themselves which of their submissions they want to make public.

With kind regards, 

Maria F.

---

From: axolo...@googlegroups.com <axolo...@googlegroups.com> on behalf of Andrey Kutuzov <and...@ifi.uio.no>
Sent: 12 April 2024 23:55:35
To: axolo...@googlegroups.com
Subject: Re: [axolotl] per-language results

 

To unsubscribe from this group and stop receiving emails from it, send an email to axolotl-24+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/axolotl-24/dc1d7e2d-067d-45da-aa1c-a8ad85166f7b%40ifi.uio.no.