Call for Participation: ClinSkill QA Shared Task @ BioNLP ACL 2026

[Paul Thompson]

We are pleased to announce the ClinSkill QA shared task, co-located with the BioNLP workshop at ACL 2026.

TASK HOMEPAGE: https://whunextgen.github.io/ClinicalskillQA/

INTRODUCTION

Multimodal large language models (MLLMs) have the potential to support clinical training and assessment by assisting medical experts in interpreting procedural videos and verifying adherence to standardized workflows. Reliable deployment in these settings requires evidence that models can continuously interpret students’ actions during clinical skill assessments, which underpins MLLMs’ understanding of clinical skills. Systematically evaluating and improving MLLMs’ understanding of clinical skills and their continuous perception in clinical skill assessment scenarios is therefore essential for building reliable and high-impact AI systems for medical education. To address this need, the shared task on medical question answering targets clinical skill assessment scenarios.

IMPORTANT DATES

Release of task data : Jan 30, 2026

Paper submission deadline: Apr 17, 2026

Notification of acceptance: May 4, 2026

Camera-ready paper due: May 12, 2026

BioNLP Workshop Date: July 3 or 4, 2026

Note that all deadlines are 23:59:59 AoE (UTC-12).

TASK DEFINITION

ClinSkill QA formulates clinical skill understanding and continuous perception for clinical skill assessment as an ordering task: the MLLM is required to arrange shuffled key frames into a coherent sequence of clinical actions and to provide explanations for the resulting order. The dataset is constructed from video clips of medical student clinical procedures, collected from Zhongnan Hospital of Wuhan University and cofun (http://www.curefun.com/). This study was approved by the Institutional Review Board (IRB), and all data collection and processing followed relevant ethical guidelines.

DATASET

ClinSkill QA is built on 200 sets of shuffled key frames extracted from three types of clinical skill videos. Each set of key frames represents a sequence of continuous actions and is accompanied by expert-annotated ground-truth ordering and order rationales.

EVALUATION

For evaluation, we use Task Accuracy (exact ordering) and Pairwise Accuracy (the fraction of adjacent pairs correctly ordered) for the ordering results, and BertScore as well as an LLM-as-judge(G-Eval) for assessing the quality of the ordering explanations.

For the i-th sample (a set of shuffled keyframes):

Ordering evaluation

- Task Accuracy

- Pairwise Accuracy

Rationale evaluation

- BertScore

-LLM-as-Judge（G-Eval）

REGISTRATION AND SUBMISSION

Registration and Submission will be done via CodaBench (Link will be available soon on the task home page)

Each team is allowed up to ten successful submissions on CodaBench.

All shared task participants are invited to submit a paper describing their systems to the Proceedings of BioNLP 2026 (https://aclweb.org/aclwiki/BioNLP_Workshop) at ACL 2026 (https://2026.aclweb.org/).

Papers must follow the submission instructions of the BioNLP 2026 workshop (https://aclweb.org/aclwiki/BioNLP_Workshop).

ORGANIZERS

Xiyang Huang, School of Artifical Intelligence, Wuhan University

Yihuai Xu, School of Artifical Intelligence, Wuhan University

Zhiyuan Chen, School of Artifical Intelligence, Wuhan University

Keying Wu, School of Artifical Intelligence, Wuhan University

Jiayi Xiang, School of Artifical Intelligence, Wuhan University

Buzhou Tang, School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen

Renxiong Wei, Zhongnan Hospital of Wuhan University

Yanqing Ye, Zhongnan Hospital of Wuhan University

Jinyu Chen, Zhongnan Hospital of Wuhan University

Cheng Zeng, School of Artifical Intelligence, Wuhan University

Min Peng, School of Artifical Intelligence, Wuhan University

Qianqian Xie, School of Artifical Intelligence,Wuhan University

Sophia Ananiadou, Department of Computer Science, The University of Manchester

--

Paul Thompson
Research Fellow
Department of Computer Science
National Centre for Text Mining
Manchester Institute of Biotechnology
University of Manchester
131 Princess Street
Manchester
M1 7DN
UK
http://personalpages.manchester.ac.uk/staff/Paul.Thompson/