[CFP] LLM as a Judge?: From Statute Prediction to Sycophancy Detection in Law

[Kripa Ghosh]

** Apologies for cross-posting **

We are glad to announce the track LLM as a Judge?: From Statute Prediction to Sycophancy Detection in Law with FIRE 2026.

Website: https://sycolex.com/

Problem Statement:

AI and Law have rapidly grown as an area of research worldwide, across the judiciary. Researchers across multiple legal systems, tasks, and languages have explored the applicability of LLMs to this field. Nonetheless, the AI prediction/recommendation system is perhaps inconsequential unless the legal experts find it useful, and, more importantly, interpretable and reliable. To provide a testbed for a robust evaluation of LLMs in AI and Law, we propose two tasks: 

1) Explainable Statute Prediction: 

In this task, we present an empirical platform for evaluating not only the predictions but also the underlying explanations acceptable to legal experts within the classical legal AI paradigm of Statute Prediction. 

So, in this task, given a set of training triplets (facts, predicted statute, explanation), participants will be given a test set of facts only, from which they will have to predict the statute and its explanation. 

2) Detection of Sycophantic Behavior in LLMs: 

Sycophancy in Large Language Models (LLMs) is the tendency to echo user beliefs regardless of truth. This poses serious risks in legal contexts, where AI-endorsed misinformation can mislead decision-making. 

So, in this task, given a legal query, the participants have to detect sychophantic behaviors using LLMs. We will have legal questions that incorporate clear outcome expectations for the training set. For the task, participants will have to predict the outcome based on the query. 

Please register here: https://sycolex.com/pages/registration.html

Organizers:

Kripabandhu Ghosh, Indian Institute of Science Education and Research, Kolkata (IISER-K), India

Liana Ermakova, Université de Bretagne Occidentale (UBO), France 

Shuvam Banerjee Seal, Indian Institute of Science Education and Research, Kolkata (IISER-K), India

Subinay Adhikary, Indian Institute of Science Education and Research, Kolkata (IISER-K), India

Jaap Kamps, University of Amsterdam (UvA), The Netherlands

Kind Regards,

Kripabandhu Ghosh 

Associate Professor,

Department of Computational and Data Sciences (CDS),

Indian Institute of Science Education and Research (IISER) Kolkata

Mohanpur, Nadia-741246, West Bengal
INDIA

Adjunct Faculty,

Dhirubhai Ambani Institute of Information and Communication Technology (DA-IICT),

DA-IICT Road, Gandhinagar-382 007, Gujarat 

INDIA

Adjunct Professor,

West Bengal National University of Juridical Sciences (WBNUJS),

12, LB Block, Sector III, Salt Lake City Kolkata – 700106

INDIA

PE   kripa...@gmail.com

OE   kripa...@iiserkol.ac.in

M   + 91 833 504 5897