[Shared task] LLM as a Judge?: From Statute Prediction to Sycophancy Detection in Law
Liana ERMAKOVA
AI and Law have rapidly grown as an area of research worldwide, across the judiciary. Researchers across multiple legal systems, tasks, and languages have explored the applicability of LLMs to this field. Nonetheless, the AI prediction/recommendation system is perhaps inconsequential unless the legal experts find it useful, and, more importantly, interpretable and reliable. To provide a testbed for a robust evaluation of LLMs in AI and Law, we propose two tasks:
1) Explainable Statute Prediction:
In this task, we present an empirical platform for evaluating not only the predictions but also the underlying explanations acceptable to legal experts within the classical legal AI paradigm of Statute Prediction.
So, in this task, given a set of training triplets (facts, predicted statute, explanation), participants will be given a test set of facts only, from which they will have to predict the statute and its explanation.
2) Detection of Sycophantic Behavior in LLMs:
Sycophancy in Large Language Models (LLMs) is the tendency to echo user beliefs regardless of truth. This poses serious risks in legal contexts, where AI-endorsed misinformation can mislead decision-making.
So, in this task, given a legal query, the participants have to detect sychophantic behaviors using LLMs. We will have legal questions that incorporate clear outcome expectations for the training set. For the task, participants will have to predict the outcome based on the query.
Kripabandhu Ghosh, Indian Institute of Science Education and Research, Kolkata (IISER-K), India
Liana Ermakova, Université de Bretagne Occidentale (UBO), France
Shuvam Banerjee Seal, Indian Institute of Science Education and Research, Kolkata (IISER-K), India
Subinay Adhikary, Indian Institute of Science Education and Research, Kolkata (IISER-K), India
Jaap Kamps, University of Amsterdam (UvA), The Netherlands