Codalab scorer giving different results than scorer script for Task 3 subtask B

[Naman Goyal]

Hi,
I tried to submit my file in the zipped format to the codalab for developement set and I keep getting following results:

MAP:0.713529909927
AvgRec:0.861103923582
MRR:76.6666666667
(which is IR baseline)

but the same file gives following result with released scoring script:
IR SYS
MAP : 0.7135 0.7259
AvgRec: 0.8611 0.8944
MRR : 76.67 79.17

Can you please suggest as why is there this difference?

Thanks,
Naman

[Naman Goyal]

Hi,

It'd be great if the organizers (/ anyone) can help me in this issue. I improved my system further and the release script gives better score but codalab still gives same output (IR baseline).

I have no idea how can I proceed in this case.

Thanks,

Naman

[Preslav Nakov]

Hi Naman,

Thanks for the note.

It looks like the scorer reports the IR baseline results, not the system ones, and also with more decimal points. We are investigating this and we will post a fix.

Preslav

[Naman Goyal]

Hi Preslav,

Thanks for responding.

I am looking forward to resolution of this.

[Preslav Nakov]

Hi Naman,

This has been fixed a few days ago.

Can you try again?

Preslav

--
Task website: http://alt.qcri.org/semeval2016/task3/
---
You received this message because you are subscribed to the Google Groups "SemEval-2017 Task 3 CQA" group.
To unsubscribe from this group and stop receiving emails from it, send an email to semeval-cqa+unsubscribe@googlegroups.com.
To post to this group, send email to semev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/semeval-cqa/d7818bd9-3f0c-483d-a3cc-1d722295ff2b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Naman Goyal]

Hi Preslav, 

I just tried this again but it still doesn't work for me. 

The command I use on my machine to get score is following:

python MAP_scripts/ev.py SemEval2017-Task3-CQA-QL-dev.xml.subtaskB.relevancy SemEval2017-Task3-CQA-QL-dev.xml.subtaskB.pred

And I get:

*** Official score (MAP for SYS): 0.7455

But still codalab gives IR baseline. Am I doing some mistake here? The submitted zip file has just one file "SemEval2017-Task3-CQA-QL-dev.xml.subtaskB.pred".

Thanks,

Naman

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-cqa...@googlegroups.com.

To post to this group, send email to semev...@googlegroups.com.

[Naman Goyal]

Hi Preslav,

Can you please suggest as what should I do about this issue?

Is it something only I am facing due to some mistake? I can share my pred file also if that's helpful.

It'd really help a lot if you can suggest what mistake I might be doing here in submission. As other than that I might not even be able to make any submission.

Thanks,

Naman 

[Luis Marquez]

Hi Naman,

I have also submitted several runs to Task B to check, and apparently it works well. When I evaluate locally an output file I get the same scores as the ones that are displayed in the leaderboard. I cannot reproduce the strange outcome you are describing... Any one in the group is experiencing the same problem as Naman?
@Naman, please send me the file and I'll try to upload it myself.

Regards,
Lluís

On Sunday, January 8, 2017 at 2:36:49 AM UTC+3, Naman Goyal wrote:

Hi Preslav,

[asma.be...@gmail.com]

Hi all,

I submitted a run on CodaLab to check and I got the same results as my local tests using the scorer.

@Naman: As the file to upload should be named SemEval2017-Task3-CQA-QL-dev.xml.subtaskB.pred, maybe you uploaded the file provided by the organizers with the results of the baseline system and not the file with your results. 

Best,

Asma

[Naman Goyal]

Hi Asma,

Thanks for responding.

I actually shared my pred file with the organizers and it seems they can reproduce my issue. So it seems like there's some difference in local scorer vs codalab scorer in terms of file format or something. 

I am sure that I am submitting SemEval2017-Task3-CQA-QL-dev.xml.subtaskB.pred as if the file is not present by the same name, codalab throws an error.

Thanks,

Naman

--

Thanks,

Naman

[Preslav Nakov]

Indeed, Naman's file yields a strange result with the CodaLab scorer, while that scorer works fine with any other input we have tried it with. We are investigating the reasons for this.

Naman's file is scoring fine with the offline scorer v2.3.

While we are investigating the issue, participants should not be worried as we will not rely on scoring in CodaLab only, and we will also score the submissions with the released offline scorer, to make sure there are no issues. After the submission deadline, we will also release the gold labels, so that participants can do scoring themselves and double-check their scores.

Preslav

--

Task website: http://alt.qcri.org/semeval2016/task3/
---
You received this message because you are subscribed to the Google Groups "SemEval-2017 Task 3 CQA" group.

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-cqa+unsubscribe@googlegroups.com.

To post to this group, send email to semev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/semeval-cqa/CAJ9Tq%2Bv9SFMp%3DD4%2BaE1kgnkyspDoppFSHUAoa2%3DaaPGwisxtVA%40mail.gmail.com.

[Doris Hoogeveen]

Hello Naman,

Preslav asked me to look into the issue. I think I have found the problem. Could you please test in Codalab to see if you get the same output as with the offline script now?

Doris

[Naman Goyal]

Hi Doris,

Awesome, it is resolved now. I can see correct score on codalab. 

Thanks a ton for looking into it.

Thanks,

Naman

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-cqa...@googlegroups.com.

To post to this group, send email to semev...@googlegroups.com.