Format submission

[Kevin Kloos]

Dear all,

I have two questions regarding the submission of Task T1A of the LeQua2022 competition.

The first question is about the number of evaluated test sets. The format says the following:  "There should be exactly 1000 rows (for tasks T1A and T2A) or exactly 5000 rows (for tasks T1B and T2B), after the header.".  However, the folder with test sets of Task T1A on Zenodo (T1A.test.zip) seems to contain 5000 files (contrary to the development samples which contain 1000 files). Should we submit the first 1000 test sets only or all 5000?

The second question is about the file format of the submission. CodaLab requires the following: "CodaLab requires each submission file to be called exactly "answer.csv" and be compressed as a zip file (the zip file needs not be called "answer.zip" though)." All evaluation examples are performed with .txt files. Should we transform the .txt-file to a .csv-file (and then zip the file)?

Last, thank you for organizing this lab. It was fun to do and it gave me a lot of research inspiration!

Thank you in advance and kind regards,

Kevin Kloos

[Alex Moreo]

Dear Kevin,

The test sets consist of 5000 samples in all cases (i.e., T1A, T1B, T2A, T2B), so you should provide a result file containing 5000 lines (after the header). See Section 3.3. in the official LeQua2022 task proposal (https://arxiv.org/pdf/2111.11249.pdf). But you are perfectly right, there was an error in the format specification (sorry for that!). We have already corrected it; thanks for spotting it! 

Regarding your second question: yes, CodaLab requires you to name your result file exactly "answer.csv", and then zip it. The following snippet should do the trick:

import zipfile

def dozip(filein):

     zipout=filein.replace('.txt', '.zip')

     with zipfile.ZipFile(zipout, 'w') as myzip:

          myzip.write(filein, arcname='answer.csv')

Best regards!

Alejandro

[Kevin Kloos]

Dear Alejandro,

Thank you for your fast and elaborate reply! The snippet did the work and I have submitted the file.

Best regards,

Kevin