LLMs4OL Codalab Competition Page is Now UP

[LLMs4OL Challenge]

Dear Participants,

We hope you are doing well,

We are approaching the evaluation stage, and we are happy to share the Codalab competition page (https://codalab.lisn.upsaclay.fr/competitions/19547) to receive the predictions for evaluation sets. The Codalab page consists of the submission format, evaluation, and all the necessary details for the submission. 

We will have practice sessions until 27 th of June and then we will proceed according to the schedule that we had previously as follows:

• June 27 2024: Few-shot and zero-shot test datasets release for evaluation stage 1 (Task A)
• June 28 2024: Begin accepting system submissions for evaluation stage 1 (Task A)
• July 04 2024: End accepting system submissions for evaluation stage 1 (Task A)
• July 05 2024: Few-shot and zero-shot test datasets release for evaluation stage 2 (Tasks B & C)
• July 06 2024: Begin accepting system submissions for evaluation stage 2 (Tasks B & C)
• July 18 2024: End accepting system submissions for evaluation stage 2 (Tasks B & C)
• July 28 2024: Participant papers submission due
• August 18 2024: Notification to authors
• August 30 2024: Camera-ready due
• November 2024: ISWC 2024, Baltimore, Maryland, USA

We made one specific exception to ease your submission by accepting in total of 30 submissions per subtask and leaving submission for stage 1 open even after submission for stage 2 starts.

Also please find the submission format at (https://codalab.lisn.upsaclay.fr/competitions/19547#learn_the_details-submission) and the evaluation criteria at (https://codalab.lisn.upsaclay.fr/competitions/19547#learn_the_details-evaluation). At the start of each phase, you can find the blind test dataset at the "Participate>Files" section of the Codalab, for both zero-shot and few-shot datasets (the only exception is for UMLS test sets, which we will share privately with participants that obtained the license). 

Due to the large number of subtasks, the practice phase is designed to help you prepare your submission for tasks and it will end on June 27, 2024. In the practices, we accept submissions by considering train sets as evaluation sets for Task A (only for WordNet subtask), Task B (only for GeoNames subtask), and Task C (only for GO task). The desirable format for practice can be obtained from the "Participate>Files" page.

Tasks:

• Task A: For a given term, you need to identify the types. (Note: For few-shots usually these types are the same as you saw in the train set. and for zero-shot, we will provide the terms and set of unique types that you need to assign a type to a term)
• Task B: For a given set of types, you need to identify the (parent, child) pair that is shaping an "is-a" taxonomy between types. 
• Task C: For a given set of types and relations, you need to identify the (h, r, t) triplet that is shaping non-taxonomic relations. 

Important‌ Links:

• LLMs4OL CodaLab: https://codalab.lisn.upsaclay.fr/competitions/19547
• Submission format for tasks: https://codalab.lisn.upsaclay.fr/competitions/19547#learn_the_details-submission
• Evaluation Criteria https://codalab.lisn.upsaclay.fr/competitions/19547#learn_the_details-evaluation
• Public Data: https://codalab.lisn.upsaclay.fr/competitions/19547#participate-get_starting_kit 
• Github page: https://github.com/HamedBabaei/LLMs4OL-Challenge-ISWC2024 

Please do not hesitate to contact us if you have any issues with Codalab or questions about the submissions. 

Best Regards,

Hamed,

On behalf of the LLMs4OL Organization Team.

[jenlindadsouza]

Dear participants,

we would kindly request and strongly recommend you make practice submissions to ensure that your submissions in the test phases can be smoothly accepted. The practice phase is still running (you can find them as the yellow marked tabs on Codalab). Please do let us know if you have any questions. We are here to offer any support or clarifications as best possible.

Warm regards,

Jennifer

[Anonymity Klein]

Hi there!

Thanks for all you have prepared! If i understood it correctly, right now it should be the time to release the test datasets for Task A. But unfortunately i don't see the files could be downloaded for taskA. The files provided are still for the first practice phase. 

Maybe I should come back later?

BR,

Yixin

[Jennifer D'Souza]

Dear Yixin,

yes you are right. The files will be released by midnight today. Please do indeed check back.

Kind regards,

Jennifer

--
You received this message because you are subscribed to the Google Groups "LLMs4OL Challenge" group.
To unsubscribe from this group and stop receiving emails from it, send an email to llms4ol-challe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/llms4ol-challenge/a844b715-1fab-4ad3-9c91-d568bcf9111cn%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pankaj Goyal]

Dear sir, i also request you to please increase the deadline of task A test to atleast till 7th or 8th july if possible so that we have enough time to try more and more things and to increase our accuracy.

To view this discussion on the web visit https://groups.google.com/d/msgid/llms4ol-challenge/CACN9A1WiBi05x4AycaboK%3Db8Cuyp3Yh3aaZJcuz4KsDtnv7R1Q%40mail.gmail.com.

[LLMs4OL Challenge]

Dear Pankaj,

Thanks for your interest. We already made this exception to ease your submission by accepting 30 submissions per subtask and leaving submission for stage 1 (Task A evaluation) open even after submission for stage 2 starts (Task B & C evaluations). So for task A, you will have time until 18 July. But to make the submissions fair we set a limit of 30 submissions per subtask. 

We hope this helps.

Best Regards,

Hamed,

On behalf of the LLMs4OL Organization Team.

To view this discussion on the web visit https://groups.google.com/d/msgid/llms4ol-challenge/CALbhffmcoMYCCbZEnYS2nDxBQCHijhSNAwV_XhDDnb-GvGWfNg%40mail.gmail.com.

[Carick Atezong]

Hello,

Please, for the moment, on Codalab, do we only need to submit the JSON files of our model predictions obtained with the training data for Task A? Is that correct, please ?

Thank you in advance for your clarification.

[jenlindadsouza]

Hi there,

We have now started the evaluation phase for Task A. Please refer to my most recent email for the official announcement.

To address your question, the practice phases were purely optional and were designed to help participants get familiar with making their submissions in the correct format on Codalab. This way, you won’t lose time figuring it out during the evaluation phase.

I hope this helps!

Kind regards,
Jennifer