[Advertisement-in-Retrieval-Augmented-Generation] Test Datasets

[Sebastian Heineking]

Hi all,

we want to give you an update on the test sets for the two sub-tasks of the Advertisement in Retrieval-Augmented Generation Task.

Submission Process
The submissions are made via TIRA. Please follow the instructions under "TIRA Tutorial and TIRA Baselines" as mentioned in our previous mail.
The page for our task is https://www.tira.io/task-overview/advertisement-in-retrieval-augmented-generation-2025/. Here, you can submit your approaches to both sub-tasks as explained under https://docs.tira.io/participants/participate.html and illustrated in the READMEs of our baselines.

Sub-Task 1 - Generation:
To enable the use of LLM-APIs, we have made the test set for this task publicly available via Zenodo. This allows you to create your runs locally and submit the run files to TIRA.
Both locally created runs as well as code submissions are made via TIRA to the dataset "ads-in-rag-task-1-generation-test".

Notes on the Zenodo-Dataset:

• The file ads-in-rag-task-1-generation-test.jsonl.gz contains a list of queries with candidate segments and advertisements (refer to our webpage for an example).
• The corresponding "labels"-file contains reference data in the correct output format. Please ensure that your output has the same format.
  As outlined under submission, your approach needs to create a new row for each combination of query and element from the list of advertisements. The first row for each query contains the response without an ad and has the ID-suffix "-0".

Sub-Task 2 - Classification:

The test dataset for the classification task is available via TIRA under the name "ads-in-rag-task-2-classification-test".

Please use the code submission for this task as explained in the TIRA documentation.

Best regards,
Sebastian

--
Sebastian Heineking

Leipzig University
Intelligent Language Technologies
Augustusplatz 10, 04109 Leipzig

sebastian...@uni-leipzig.de