Submissions not recorded?

[Dirk]

In my participate -> post-evaluation page, I notice that only submissions up to 30th Jan 2022 are counted for the scores. All submissions afterwards don't have any scores. Is this a bug/error?

[Pisarevskaya Dina]

The same issue here, too.

четверг, 3 февраля 2022 г. в 00:36:23 UTC, dirk.l...@gmail.com:

[Rishikesh Devanathan]

Dear Organizers,

We also have the same issue. Although submissions after 30th Jan were recorded for us, only some of them were included. We submitted more than 9 models but it evaluated for only 9. Please rectify this as soon as possible.

[Famille Dufour]

Dear Organizers,

Same issue. 

another question: will it be possible to receive Overall scores by peer id of the real annotations of the evaluation. Just to understand where we made important gaps and how to improve our model (scraping problems, did we have to translate the set into English, choice of algorithms, data train imbalance)

best regards

[Михаил Сергеевич Куимов]

The same

3 февр. 2022 г., в 10:54, Famille Dufour <dufou...@gmail.com> написал(а):

Dear Organizers,

--
You received this message because you are subscribed to the Google Groups "semeval-2022-task-8-multilingual-news" group.
To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2022-task-8-mult...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/semeval-2022-task-8-multilingual-news/aec85bd1-a9d0-421a-a608-ee33e4c8bb17n%40googlegroups.com.

[Dirk]

I noticed on leaderboard that some of the scores are from submission after 30.01.22.

However, my submissions after 30.01.22 are still not scored yet. The predictions from our final models were submitted all during those period (after 30.01.22).

Could anyone from the organizer team check out this issue please?

Thank you.

Dirk

[Scott Hale]

Hi All,

The leaderboard contains only one score per user/team. Prior to 30 Jan., it was sent to replace the current score with the new score regardless of whether it was higher or lower. This meant that if you submitted a worse performing model you would move down the leaderboard. On 30 Jan, we changed the setting to keep the highest score. Since that time, a new score would go on to the leaderboard only if it was better than all previous scores. We re-ran all submissions on that day as well to ensure that all teams were treated the same and the final leaderboard had the highest score from all participants. 

We're double-checking everything and will release the final results of the competition next week.That said, we encourage everyone to submit a paper regardless of your rank. If you do not submit a paper, it would still be fantastic if you open-source your code. We'll circulate a form to collect URLs and start to build critical mass in this area.

Best wishes,

Scott

To view this discussion on the web visit https://groups.google.com/d/msgid/semeval-2022-task-8-multilingual-news/a9012702-719d-40e2-9e80-09d21cf9d5f8n%40googlegroups.com.

--

Dr Scott A. Hale

http://scott.hale.us
computer...@gmail.com

[Dirk]

Hi Scott,

I understand how it works. It's not that I think that our final models would perform better so it should be in the leaderboard to improve our rankings. The reason I said before that I notice that our submissions after 30.01.22 weren't being scored is because in the entries column, it only says 17 entries (which is also the same number as in the Participate->Submit/View Results->POST-Evaluation). However, in the Participate->Submit/View Results->EVALUATION page, it can be seen that from 31.01.22 to 01.02.22, there were additional 7 new entries. So, even if those entries doesn't make it to the leaderboard, the number in the entries column should be 24 instead of 17.

That's why I insisted on saying that some submissions haven't been included for the scoring. Please check this issue.

Thank you.

Dirk

[Scott Hale]

Hi Dirk,

Submissions were not "auto migrated" to the post-evaluation phase, but all submissions in the evaluation phase that have a finished status were processed and will be reflected in the final leaderboard for the evaluation phase.

Best wishes,

Scott

To view this discussion on the web visit https://groups.google.com/d/msgid/semeval-2022-task-8-multilingual-news/e96a3718-036f-4512-a6c4-828ba92e3314n%40googlegroups.com.

[Dirk]

So even though there are 7 more submissions, but since they are not 'migrated' to the post-evaluation phase, there's no way that we can see the scores for these 7 submissions (the 17 submissions that were 'migrated' all have their own scores in the view results page, even though they don't get into the leaderboard)?