AXOLOTL test phase started

[Andrey Kutuzov]

Dear AXOLOTL'24 participants,

The test phase of our shared task has started now.

You can find the test sets here:
https://github.com/ltgoslo/axolotl24_shared_task/tree/main/data/test

There are three test languages:
- Finnish
- Russian
- German

For Finnish and Russian, you already had a chance to try tour systems
with the training and development data
(https://github.com/ltgoslo/axolotl24_shared_task/tree/main/data).
German is a "surprise language". We will not hide its origin: it comes
from the DWUG DE Sense dataset
(https://www.ims.uni-stuttgart.de/forschung/ressourcen/experiment-daten/dwug-de-sense/),
which can be already familiar for some of you.
However, we ask you to refrain from using the original dataset to
produce predictions for AXOLOTL.
Let's make it a bit more interesting and test whether your systems are
really cross-lingual!

=============
We remind that the deadline for submitting predictions is April 9, 2024.
=============

The data format of the test sets is the same as in the training and
development sets, but the `sense_id` and `gloss` fields are empty for
the usages from the new time period.

The task of the participants is to fill in these values (senses for
Subtask 1 and definitions for the novel senses for Subtask 2).

- Codalab competition for Subtask 1:
https://codalab.lisn.upsaclay.fr/competitions/18009
- Codalab competition for Subtask 2:
https://codalab.lisn.upsaclay.fr/competitions/18008

Please strictly follow the correct format of predictions both for
Subtask 1 and for Subtask 2. We prepared toy submissions for you to make
sure you are using the right format:
https://github.com/ltgoslo/axolotl24_shared_task/blob/main/data/sample_predictions_track1.tsv

https://github.com/ltgoslo/axolotl24_shared_task/blob/main/data/sample_predictions_track2.tsv

Note also that today we fixed a minor inconsistency in our evaluation
script for Subtask 2. It influenced BLEU scores only. Both the AXOLOTL
Github repository and the Codalab competition are now updated, and we
hope this fix will not influence your submissions.

We are looking forward to your submissions!

On behalf of other organizers:

--
Andrey
Language Technology Group (LTG)
University of Oslo