Test data for subtasks A, B, C, and D

[Luis Marquez]

Dear participants,

the test sets for subtasks A, B, C, and D are now available. Get them at:

      http://alt.qcri.org/semeval2017/task3/index.php?id=data-and-tools

Please visit the following page to learn how to submit results:

     http://alt.qcri.org/semeval2017/task3/index.php?id=submitting-results

The evaluation period will end on the 30th of January.

Recall that the test set for task E will be released on January 21, 2017.

Regards,
Task 3 organizers

[Yassine El Adlouni]

Hi Luis,

Is there some a time span between downloading de test set and the submission date?

If I downloaded the test dataset yesterday, do I need to submit results after one week or the deadline is the 30th January no matter when I downloaded it?

Thanks,

Yassine

UPC-USMBA team (Task 3 Subtask D).

--
Task website: http://alt.qcri.org/semeval2016/task3/
---
You received this message because you are subscribed to the Google Groups "SemEval-2017 Task 3 CQA" group.
To unsubscribe from this group and stop receiving emails from it, send an email to semeval-cqa+unsubscribe@googlegroups.com.
To post to this group, send email to semev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/semeval-cqa/eb797e4f-ab6a-4f62-b012-9725f09daf9c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Lluis Marquez]

No constraints. You can download the dataset right away. The deadline for submitting results is invariant (January 30).

Regards,

Lluís

[Yassine El Adlouni]

Great, thanks.

Best regards,

Yassine

[Luis Marquez]

Dear participants,

we have introduced a correction in the README file of the test distribution for subtasks A-D.
The number of output predictions expected from the scorer for subtask D is 12,600, instead of the initially indicated 9,589.

Thanks to the participants for pointing us to this mistake.

Regards,
The organizers of Task 3

[Preslav Nakov]

The deadline is always 30th, regardless of when you have downloaded the data.

Preslav

To view this discussion on the web visit https://groups.google.com/d/msgid/semeval-cqa/CALX3%2Bob2NmApvGH-isJ9-77nKO6%2B5KP%3D_avndNoXJF35K_YGHw%40mail.gmail.com.

[Yassine El Adlouni]

Hi,

I've tried to submit for the dev dataset (Task 3 Subtask D) but the submission has failed.

Looking into the scoring output log, it seems that it's related to the  ID mismatch error: 

ERROR: ID mismatch on line 1:
in /codalabtemp/tmpM8_4qm/run/input/ref/SemEval2017-Task3-CQA-MD-dev-subtaskD.xml.relevancy we have (200427,9194),
but in /codalabtemp/tmpM8_4qm/run/input/res/SemEval2017-Task3-CQA-MD-dev-subtaskD.xml.pred we have (200427,319893)

I've tried both the sequence of the original XML file and an alphabetically sorted one but failed in both cases.

Could you precise please the sorting rules for the predictions file?

Best,

Yassine

UPC-USMBA Team

ᐧ

[Preslav Nakov]

Hi Yassine,

Indeed, the sorting is the problem.

The comments in each question are to be sorted by comment ID:

69579   43523

69579   51753

69579   61846

69579   63972

69579   117912

69579   229172

69579   289894

69579   684584

69579   880238

69578   47192

69578   52302

69578   86270

69578   304444

69578   432499

69578   561870

69578   696681

69578   756956

69578   875015

Preslav