[Announcement] Test Set Release, Important Updates on Submissions, Evaluation, and Timeline

[LLMs4OL Challenge]

Dear Participants,

We are sharing important updates regarding the challenge. Including the submission procedure, evaluation setup, and timeline.

Test Sets: Test cases are available now at the challenge GitHub repository: https://github.com/sciknoworg/LLMs4OL-Challenge/tree/main/2026. The expected output formats and more details are available at https://sites.google.com/view/llms4ol2026/submission.

Submission Procedure: Based on experience from previous editions, we observed that leaderboard-based platforms can sometimes shift attention toward incremental score improvements rather than the broader scientific goals of the challenge. To better support reproducibility and methodological progress, we will use a simplified submission setup:
- No CodaBench and no public leaderboard
- Submissions will be collected via a Google Form
- Test sets will remain fixed throughout the challenge

Evaluation Setup:
- Each team may submit up to 5 runs per task until 17 June 2026
- After this date, we will conduct a private intermediate evaluation for the submitted 5 runs.
- Results will be shared privately with each team for feedback and system improvement.
- After 17 June, each team may submit  2 final runs per task as a final submission.

- Final evaluation will be based only on these final submissions
- The same test sets will be used for both intermediate and final evaluation

Reporting Issues: If you identify any potential issues in the test cases, please report them by providing: task name, test case ID, and a brief description of the issue.  All reports will be carefully reviewed. While the datasets have been thoroughly checked, minor issues may still remain.

Timeline Updates
* Submissions open: June 2, 2026
* Intermediate submission deadline: June 17, 2026
* Final submission deadline: June 30, 2026
* System integration phase: July 1–19, 2026
* Paper submission deadline: July 19, 2026

Submission Form: The submission form will be announced in a separate communication and will be updated at https://sites.google.com/view/llms4ol2026/submission

We thank you for your participation and look forward to your contributions. Meanwhile, if you have any questions, please don't hesitate to reach out.

Best Regards,

Hamed Babaei Giglou

Jennifer D'Souza

Sören Auer

LLMs4OL 2026 Challenge Organizing Team

[LLMs4OL Challenge]

Dear Participants,

The submission form for the LLMs4OL 2026 Challenge is now open: https://forms.gle/cgLWLhj6zzZTTayj6 (refer to the form for further details)

Furthermore, please be aware that:

• Due to unforeseen technical issues, the evaluation scripts may be delayed by a few days. In the meantime, please refer to OntoLearner and the task pages for evaluation details. The metrics will be the same as those embedded in OntoLearner, with additional metrics.
• Submit each run separately as a ZIP file (one run per ZIP).
• Limits: up to 5 runs per task (intermediate) and 2 runs per task (final). Only the most recent submissions within the limit will be considered.

Please let us know if you face any issues with submitting your runs or accessing the Google Form. Thank you!

Best regards,
LLMs4OL 2026 Organizing Committee