Important Update on Final Evaluations, Paper Submission, and Challenge Timeline
LLMs4OL Challenge
Dear LLMs4OL 2026 Challenge Participants,
We would like to thank all teams for their participation in this year's LLMs4OL Challenge and for your patience throughout the final evaluation process.
We would also like to acknowledge that several milestones in this year's challenge timeline required adjustments compared with the original schedule, including resource releases, submission periods, and evaluation timelines. We understand that these changes may have created challenges for teams when planning their submissions and preparing their papers. We sincerely appreciate your patience, flexibility, and continued engagement throughout the process. These adjustments were made to ensure that participants had access to the necessary resources, that submissions could be properly evaluated, and that the final results are accurate and fair.
Final Leaderboard: The release of the final leaderboard has been delayed due to the additional time required to verify reported results and resolve inconsistencies in a small number of submissions. Our priority is to ensure that all submitted systems comply with the challenge rules and that the final rankings accurately reflect the performance of participating teams. As a result, the official leaderboard will be released after the paper submission deadline. This additional verification is necessary to ensure the fairness, integrity, and reproducibility of the competition results. We appreciate your understanding as we complete this verification process.
- Small note on metrics: The final ranking will be based on averaged graph similarity for tasks A and B, and for Task C, this will be averaged F1 scores. The provided additional metrics are only extra metrics that you can use to judge your systems, and they will not be considered for final ranking.
- Side note regarding confirmations: If you received an email requesting confirmation of your submissions and haven’t responded yet, please do so as soon as possible. This will help us proceed with the evaluation process and prepare the leaderboard.
Paper Submission: Please proceed with preparing your paper regardless of the final leaderboard. All participating teams are encouraged to submit a paper describing their approach. Your manuscript should be written as a standard academic conference paper, presenting your work objectively and professionally. Please focus on describing your motivation, methodology, system architecture, implementation details, experimental setup, and analysis. Avoid including leaderboard rankings or comparisons based on the competition results, as the final rankings have not yet been officially released. While we could release a preliminary leaderboard, it may still change after final verification. To avoid potential inconsistencies and unnecessary revisions to submitted papers, we will release the official leaderboard after the paper submission deadline. After the paper review process, accepted papers will have the opportunity to include the official leaderboard ranking and corresponding comparisons in the camera-ready version. To accommodate this, the paper length will also be increased by one page.
- The EasyChair for paper submissions is: https://easychair.org/conferences/?conf=llms4ol2026
- Paper preparation guidelines with all the relevant citations that you might want to use: https://sites.google.com/view/llms4ol2026/author-guidelines
- System integration (code/notebook submission): July 26, 2026 [Unchanged]
- Paper submission: July 28, 2026
We understand that these timeline adjustments may have affected teams differently. If you have any concerns or specific questions, please feel free to contact us. We will carefully consider your feedback and do our best to support all participating teams.
Thank you once again for your patience, cooperation, and participation.
Best regards,
The LLMs4OL 2026 Organizing Committee