Call for Participant (OAEI Bio-ML track)

[Jiaoyan Chen]

Dear All, 

We are organizing a new OAEI track named Bio-ML (https://www.cs.ox.ac.uk/isg/projects/ConCur/oaei/2022/) in 2022. This track aims at evaluating not only traditional OM systems, but also machine learning-based systems. Different from previous OAEI tracks, it has quite a few new features:

1. 5 biomedical ontology pairs with reliable mappings from both UMLS and Mondo are made. They are OMIM-ORDO, NCIT-DOID, SNOMED-FMA (Body), SNOMED-NCIT (Pharm) and SNOMED-NCIT (Neoplas). 

2. Not only class equivalence mappings but also class subsumption mappings are evaluated.

3. The ontologies are extracted from the original ontology with pruning for ensuring high mapping recalls as well as ontologies of different scales.

4. Not only testing mapping sets but also validation mapping sets are provided. The validation sets are provided to adjust system parameters.

5.  A comprehensive evaluation framework is set: the overall system output mappings are evaluated by Precision, Recall and F1 score; while the (ML) systems which can predict mapping score can be further evaluated by ranking-based metrics including MRR and Hit@K, with a number of hard candidates for ranking. Subsumption mapping is only evaluated by ranking-based metrics. 

To encourage more participants, the minimum request for participating in this track is submitting the system output mappings (and/or mapping scores). We encourage participants to submit reproducible systems (or package their systems by Melt), but it is not mandatory. 

We will provide certificates for:

• Top-performer systems.

• Systems that can be easily reproduced. 

Here is a preliminary timeline: 

Result submission: 10th September

System submission (optional): 20th September

Result ranking: 5th October

System paper (optional): 15 November

Materials: 

Bio-ML homepage: https://www.cs.ox.ac.uk/isg/projects/ConCur/oaei/2022/

Resource paper with preliminary benchmarking: https://arxiv.org/abs/2205.03447

Regards,

Jiaoyan Chen (on behalf of all the Bio-ML organizers)

[Francis Gosselin]

Hi,

I have questions regarding the BioML track:

1. Is there any change on the preliminary timeline (extension for the submission)?

2.  As for the submission: What is the protocol? Do we simply add a BioML.zip file in the google form along with our system?  Also, what is the required format for the predictions (rdf/tsv)?

Regards,

Francis

[Catia Pesquita]

Dear organizers,

We would also be grateful for a deadline extension. Any chance of one?

Best regards,

Catia 

Catia Pesquita

--
You received this message because you are subscribed to the Google Groups "Ontology Alignment Evaluation Initiative (OAEI)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ontology-alignment-evaluatio...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/ontology-alignment-evaluation-initiative-oaei/205fb483-f5f1-4dce-8ecd-2ef5e340385fn%40googlegroups.com.

[Ernesto Jimenez-Ruiz]

Hi Francis and Catia

As this is the first year. Yes, there will be an extension for the BioML track.

We will also add additional guidelines for the submission.

Best

Ernesto

To view this discussion on the web, visit https://groups.google.com/d/msgid/ontology-alignment-evaluation-initiative-oaei/CAJ1Uy2XPtABYdqKfHasvM-6EBB2_Tic0U0NEBXYuLnSFT-zXWw%40mail.gmail.com.

--

Ernesto Jiménez-Ruiz
Lecturer in Artificial Intelligence

Department of Computer Science

School of Science & Technology
City, University of London

https://www.city.ac.uk/people/academics/ernesto-jimenez-ruiz

[Jiaoyan Chen]

Hi, 

To continue Ernesto’s last message, we just updated our bio-ml website (https://www.cs.ox.ac.uk/isg/projects/ConCur/oaei/2022/), with the guidance and submission instructions updated. 

Regards,

Jiaoyan