Dear Community,
We would like to invite you to take part in this year's SemTab challenge. SemTab 2025 will be hosted by the International Semantic Web Conference (ISWC 2025), to be held in Nara, Japan, from November 2 to 6, bringing together researchers and practitioners to advance the field of semantic technologies.
This year's challenge mandates the use of Large Language Models (LLMs) for all tasks. Participants are expected to employ LLM-based approaches, either through fine-tuning or Retrieval-Augmented Generation (RAG) strategies, to tackle the semantic annotation tasks. This focus aims to explore and benchmark the capabilities of LLMs in interpreting complex and realistic tabular data.
Challenge Tracks and Datasets
The 2025 challenge features two main tracks, each designed to test different aspects of semantic table interpretation:
MantisTable: This track utilises a carefully selected subset of 870 tables from the new version of the MammoTab dataset, comprising a total of 84,907 cell annotations. Participants are expected to address challenges such as disambiguation, homonymy, alias resolution, NIL detection, noise robustness, and collective inference. Only approaches based on Large Language Models are allowed, either in fine-tuning settings or using Retrieval-Augmented Generation strategies. The evaluation will focus exclusively on the Cell Entity Annotation (CEA) task.
Secu-table: This dataset involves security data extracted from Common Vulnerability and Exposure (CVE) and Common Weakness Enumeration (CWE) data sources. It comprises 1,554 tables, with 20% being error-free and 80% containing various challenges, such as ambiguity, NIL, missing context, and misspelt data. Participants are invited to utilise open-source large language models (LLMs) to address all STI tasks: Cell Entity Annotation (CEA), Column Type Annotation (CTA), and Column Property Annotation (CPA), using the SEPSES Computer Security Knowledge Graph, and the CEA task using the Wikidata Knowledge Graph.
Tentative Schedule
Release of datasets and instructions: June 9th, 2025
Round 1 submission deadline: July 9th, 2025
Note: To be invited to the conference for a presentation, you must submit to Round 1
Evaluation: from July 10th, 2025
Paper submission deadline: July 16th, 2025
Invitation to present at ISWC: July 23rd, 2025
Paper camera-ready submission deadline: September 15th, 2025
Round 2 (Final) submission deadline: October 20th, 2025
Final results to be announced at the conference: November 2-6, 2025