Test Set Evaluation

[Mikołaj Tym]

Dear LM-KBC Organizers,

I have a question about the test set evaluation policy that I would like to clarify before the submission deadline.

It turned out that up to 100 submissions are allowed against the test set, which opened for evaluation on August 8. Since then, I have noticed that some public, test leaderboard scores have risen significantly.

I wanted to ask: is adjusting approach implementation based on the score returned by previous test submissions is considered within the competition rules? Or is the test set intended to be used more like a single evaluation, reserved for a final check?

Current setup makes it hard to tell which reported scores reflect genuine generalization versus test set informed tuning.

I would also like to ask: when it comes to accepting the shared task paper, is the leaderboard score a primary factor, or is it more about the described methodology?

Thank you for your answers.

Best regards,

Mikołaj Tym

Nadawcą korespondencji i jednocześnie administratorem Państwa danych osobowych jest Uniwersytet Ekonomiczny w Poznaniu, al. Niepodległości 10, 61-875 Poznań, z którym można skontaktować się listownie lub poprzez wiadomość e-mail - ro...@ue.poznan.pl Państwa dane osobowe przetwarzane będą w celu prowadzenia niniejszej korespondencji i przez czas jej trwania. Przysługuje Państwu prawo dostępu do danych, ich sprostowania, ograniczenia przetwarzania, usunięcia, żądania sprzeciwu wobec przetwarzania oraz wniesienia skargi do Prezesa Urzędu Ochrony Danych Osobowych. Szczegółowe informacje na temat ochrony danych osobowych dostępne są w Polityce prywatności: https://ue.poznan.pl/uniwersytet/polityka-prywatnosci/

[Simon Razniewski]

Dear Mikołaj,

You are right, the test set is not intended for optimizing against it, the configuration of the server this way was a regrettable mistake.

For acceptance, the prime criterion is the interestingness of the presented approach, not its achieved score.

Best wishes,

Simon

--
You received this message because you are subscribed to the Google Groups "akbc2026-shared-task" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akbc2026-shared-...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/akbc2026-shared-task/VI0PR04MB121136CAE4AF6B9ECD27B0567DBDD2%40VI0PR04MB12113.eurprd04.prod.outlook.com.
For more options, visit https://groups.google.com/d/optout.