Evaluation phase questions

[Mateusz Klimaszewski]

Dear Organizers,

I have few questions about "Scoring Test data"/evaluation phase.

As far as I understand on Jun 10th there will be another training and test datasets for evaluation phase (according to https://competitions.codalab.org/competitions/19955#learn_the_details-data-specifications).

Am I right to assume so and if yes do you already have, or if not when there will be, any data about at least order of magnitude of those training/test datasets?

Finally, question about submission phase rules, in overall SemEval rules (http://alt.qcri.org/semeval2019/index.php?id=codalab) there is information that task can have up to 10 submission during that phase.

Since there is not such information on codalab of Task 9 (or I missed it) - do you already decided about that rule?

Best regards,

Mateusz Klimaszewski

[Sapna Negi]

Hi Mateusz,

> Dear Organizers,
> I have few questions about "Scoring Test data"/evaluation phase.
>
> As far as I understand on Jun 10th there will be another training and test datasets for evaluation phase (according to https://competitions.codalab.org/competitions/19955#learn_the_details-data-specifications).
> Am I right to assume so and if yes do you already have, or if not when there will be, any data about at least order of magnitude of those training/test datasets?

I think you mean January 10. There will be a new test dataset, the training dataset will remain the same as the one currently available. As I mentioned in the previous post, there will be a final version of training dataset coming out soon (this week) which will have a few more sentences.

> Finally, question about submission phase rules, in overall SemEval rules (http://alt.qcri.org/semeval2019/index.php?id=codalab) there is information that task can have up to 10 submission during that phase.
> Since there is not such information on codalab of Task 9 (or I missed it) - do you already decided about that rule?

No, we haven’t set a maximum limit for evaluation phase submissions. But its highly likely we will follow what is recommended by SemEval. But if you have a reason for wanting to do more than 10 evaluation phase submissions, please let us know and we will take that into consideration while making a decision.

>
> Best regards,
> Mateusz Klimaszewski
>
>
Best
Sapna