[CFP] 2nd Call for Participation - CLEF 2024 SimpleText Task4: SOTA? "Tracking the State-of-the-Art in Scholarly Publications"

[Jennifer D'Souza]

[apologies if you receive multiple copies of this call]

Dear colleagues and friends,

We are pleased to release the 2nd Call for Participation - CLEF 2024 SimpleText Task4: SOTA?

Overview: SOTA? is introduced as Task 4 in the SimpleText track of CLEF 2024. The goal of the SOTA? shared task is to develop systems which given the full text of an AI paper, are capable of recognizing whether an incoming AI paper indeed reports model scores on benchmark datasets, and if so, to extract all pertinent (Task, Dataset, Metric, Score) quadruples presented within the paper.

More info on the task website: https://sites.google.com/view/simpletext-sota/home

SOTA? will be divided into two evaluation phases:

- Evaluation Phase 1: Few-shot Testing;
- Evaluation Phase 2: Zero-shot Testing

To participate in SOTA? i.e. SimpleText Task 4 @ CLEF 2024, please register your team:

1. CLEF 2024 official registration page https://clef2024.imag.fr/index.php?page=Pages/registration.html
2. Codalab competition site: https://codalab.lisn.upsaclay.fr/competitions/16616

Note, SOTA? is organized as a new task this year under the "SimpleText - Improving Access to Scientific Texts for Everyone" initiative https://simpletext-project.com/. Please take a look at the other 3 tasks, i.e. Task 1, 2, and 3, offered by SimpleText and select one or more of those task options too if you are interested. Note that there is no interdependence of the dataset between "Task 4 - SOTA?" and the other three tasks of SimpleText.

Dates

Training and validation datasets available: Feb 1, 2024 March 13, 2024
Test data available/Evaluation starts: April 23, 2024
Evaluation ends: May 3, 2024
Participant paper submissions due: May 31, 2024
Notification to authors: June 24, 2024
Camera ready due: July 8, 2024
CLEF 2024 Workshop, Grenoble, France: 9-12 September 2024

Task Organizers

Jennifer D’Souza (TIB Leibniz Information Centre for Science and Technology - Germany)
Salomon Kabongo (L3S Research Center, Germany)
Hamed Babaei Giglou (TIB Leibniz Information Centre for Science and Technology - Germany)
Yue Zhang (Berlin Technical University, Germany)
Sören Auer (TIB Leibniz Information Centre for Science and Technology - Germany)

We look forward to having you on board!

Contact: sota.task [at] gmail.com