Questions Regarding Evaluation Metrics for the Flagship Task

[Louis]

I would like to ask the organizers to provide more detailed explanations regarding the third evaluation metrics "Graph Similarity Metrics" designed for the Flagship Task. It would be greatly appreciated if you could elaborate on the specific calculation rules, scoring criteria, weighting mechanisms, and any relevant evaluation details of this task.

[LLMs4OL Challenge]

Dear Louis,

I hope this email finds you well.

Thanks for your email and for bringing this to our attention.

This metric evaluates how similar two (gold and predicted) ontology fragments are to each other (per sample). The metric uses three criteria: Edge F1, Neighborhood Similarity, and Taxonomy Similarity. The final graph similarity is the average of the three criteria. Now I have added more details on this metric at the task page https://sites.google.com/view/llms4ol2026/flagship-task (The metric calculation is also available at: https://github.com/sciknoworg/LLMs4OL-Challenge/blob/main/2026/graph_similarity_metric.py). 

I hope this answers your questions. Please let me know if you have any questions or need assistance. 

Any feedback or suggestions are welcome.

Best Regards,

Hamed

On behalf of the LLMs4OL challenge organization team

On Wed, May 13, 2026 at 5:13 AM Louis <luoy...@gmail.com> wrote:

I would like to ask the organizers to provide more detailed explanations regarding the third evaluation metrics "Graph Similarity Metrics" designed for the Flagship Task. It would be greatly appreciated if you could elaborate on the specific calculation rules, scoring criteria, weighting mechanisms, and any relevant evaluation details of this task.

--
You received this message because you are subscribed to the Google Groups "LLMs4OL Challenge" group.
To unsubscribe from this group and stop receiving emails from it, send an email to llms4ol-challe...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/llms4ol-challenge/2f8a433f-84bc-4b7d-a672-dca9e7ab38e4n%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.