Results

[Mladen Karan]

Hello,

When could we expect the test set results to become public?

Thanks and kind regards,
Mladen

[Preslav Nakov]

We have issues with CodaLab: for some systems CodaLab gives different results compared to offline scoring (this is also an issue for other SemEval tasks this year). We are double-checking every single case, and we will release the full results shortly. Overall, we do not trust CodaLab, but we trust our offline scoring.

We will also release the scorer and the gold labels, so that everybody can check for themselves.

Preslav

--
Task website: http://alt.qcri.org/semeval2016/task3/
---
You received this message because you are subscribed to the Google Groups "SemEval-2017 Task 3 CQA" group.
To unsubscribe from this group and stop receiving emails from it, send an email to semeval-cqa+unsubscribe@googlegroups.com.
To post to this group, send email to semev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/semeval-cqa/cc28a0aa-d28d-401c-a042-9711fe6cd604%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mladen Karan]

Ok, thanks.
Looking forward to the results.

Best,
Mladen

On Friday, February 10, 2017 at 2:29:13 PM UTC+1, Preslav Nakov wrote:

We have issues with CodaLab: for some systems CodaLab gives different results compared to offline scoring (this is also an issue for other SemEval tasks this year). We are double-checking every single case, and we will release the full results shortly. Overall, we do not trust CodaLab, but we trust our offline scoring.

We will also release the scorer and the gold labels, so that everybody can check for themselves.

Preslav

On Fri, Feb 10, 2017 at 4:24 PM, Mladen Karan <karan....@gmail.com> wrote:

Hello,

When could we expect the test set results to become public?

Thanks and kind regards,
Mladen

--
Task website: http://alt.qcri.org/semeval2016/task3/
---
You received this message because you are subscribed to the Google Groups "SemEval-2017 Task 3 CQA" group.

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-cqa...@googlegroups.com.

[Preslav Nakov]

BTW, this issue was noticed for task 3 at DEV time: there is a participant who was getting different results using CodaLab and using the offline scoring. We could reproduce the issue, but we could not fix it. It could be that CodaLab messes with tabs or new lines of uploaded files, or just truncates them. We could not tell.

Preslav