SemTab 2021: Error in the CEA WD task description and leaderboard

Raphaël Troncy

unread,

Jul 23, 2021, 9:12:22 AM7/23/21

to Sem-Tab Challenge, Raphaël Troncy

Dear Ernesto, all SemTab organizers,

We noticed an error at https://www.aicrowd.com/challenges/semtab-2021/problems/cell-entity-annotation-by-wikidata-cea-wd. The submission format which is indicated is:

“Table ID”, “Column ID”, “Row ID” and “Entity IRI”

while in reality, what we have to submit is:

“Table ID”, “Row ID”, “Column ID” and “Entity IRI”

We think you should update the page!

Furthemore, we would like to know if there is any restrictions regarding the number of submissions. Is this capped or unlimited? Will this be taken into account? We do not plan to do hundreds of submissions but we would like to know if we have to pack multiple updates into runs or if we could decompose them to better assess their improvements.

Finally, can we have some visibility on all our submission results like it used to be the last 2 years? If we submit a less good performing run, we will not be able to see the scores. We cannot track the scores of our various submissions. It seems that we could retrieve only the last 3 submissions on AICrowd by guessing the URIs. Is all this intended?

Best regards

Raphaël for the DAGOBAH team

Jiaoyan Chen

unread,

Jul 25, 2021, 10:27:57 AM7/25/21

to Raphaël Troncy, Sem-Tab Challenge, Raphaël Troncy

Dear Raphael,

Thanks for pointing out this issue on “Column ID” and “Row ID”. Our original description was wrong since I did not realize the data for WD actually changed the column order as the data for DBP. But that does not impact the evaluator. I have updated the page of CEA-WD.

Currently I set the submission limit to 5 times per day per team. But if you think it is not enough, please tell me.

I just turned on the option “Show grading history”. Not sure whether this can help allow participants to see the historical results via the website.

But at least, we can export the historical submission results of all participants into an excel file. I can share that to you when necessary.

Regards,

Jiaoyan

--
You received this message because you are subscribed to the Google Groups "Sem-Tab Challenge" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sem-tab-challe...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/sem-tab-challenge/CADfgUPgg5v86%2BuVweHEmA_aeiTzE0%3DCDHpbwkAmhs4wj0VUNnw%40mail.gmail.com.

Nora Youssef

unread,

Jul 28, 2021, 5:31:22 AM7/28/21

to Jiaoyan Chen, Raphaël Troncy, Sem-Tab Challenge, Raphaël Troncy

Dear Jiaoyan,

I believe that there was a tab for "Submissions" this was really useful, would you check for this option specifically?

Moreover, "Show grading history" is not visible anywhere on my screen view.

Regards,

Nora

To view this discussion on the web, visit https://groups.google.com/d/msgid/sem-tab-challenge/E7B66417-C634-42CB-B42C-1E66A003A4E0%40gmail.com.