Questions regarding the Leaderboard for Subtasks 1 & 2

[fcas...@gmail.com]

Hello,

My team and I have two questions regarding the Leaderboard for subtasks 1 & 2.

If we submit our resutls for subtasks 1 & 2 at the same time, are the best scores for each subtask taken into consideration independently for the final ranking, or is it the submission we post on the Leaderboard? For instance, if the best result for subtask 1 is from submission 1, and the best result for subtask 2 is from submission 2, do we have to choose which submission will determine our final ranking, or otherwise both best scores are taken into account?

Also, we have a question regarding the current ranking in the Leaderboard. Does it reflect the real ranking among participants based on their submissions? During our last submission we improved our scores on both subtasks with respect to the previous one, but dropped in the ranking right after the submission.

Thank you

[Sasha Spala]

Hi!

 

Ah, I see what you’re saying about the leaderboard. Rankings for subtask 1 and subtask 2 will be computed separately, but please do put your best subtask 1 and 2 submissions together so they both appear on the leaderboard. If you run out of submission tries, please let me know. For example, if you make 5 submissions just with subtask 1, we will take your best performing subtask 1 files and zip them up for you to submit along with your subtask 2 files so that we ensure your best subtask 1 and your best subtask 2 files are accounted for on the leaderboard. We will release the final rankings and scores after the evaluation period is over.

I do apologize for the complications merging the two subtask tabs together caused – none of us were expecting these issues to pop up, but we’re doing our best to cope with them as they come.

 

What is your team name? I’ll check on your question about the leaderboard rankings.

 

Best,
Sasha

--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.
To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/semeval-2020-task-6-all/75806aab-0395-4d10-af38-b4c06b445490%40googlegroups.com.

[Fabien Caspani]

Hi Sasha,

Thank you so much for your response, it is very helpful!

In the Leaderboard the username used is "caspa", and my team name is "ACNLP".

Thanks!

Fabien

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task-6-all+unsub...@googlegroups.com.

[Sasha Spala]

It looks like there’s something wonky going on with the leaderboard ordering. I’m not sure why its ordering the way it is, but in any case just submit your best submission to the leaderboard. We will double check everything with the submissions on our end and announce the rankings and scores after the evaluation period, regardless of the state of the leaderboard ranking. In any case, SemEval does ask us to keep the leaderboard results hidden (http://alt.qcri.org/semeval2020/index.php?id=codalab) until after the evaluation phase is over.

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/semeval-2020-task-6-all/75806aab-0395-4d10-af38-b4c06b445490%40googlegroups.com.

--

You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/semeval-2020-task-6-all/b8aff464-f5cf-42eb-bb45-edc2b6c32182%40googlegroups.com.

[Fabien Caspani]

Thanks a lot!

I just wanted to make sure that something weird was going on with the current state of the Leaderboard.

Best,

Fabien

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task-6-all+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/semeval-2020-task-6-all/75806aab-0395-4d10-af38-b4c06b445490%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task-6-all+unsub...@googlegroups.com.

[Iman Saleh]

Hi Sasha, I forgot to submit the best submission to the leaderboard. Will the best submission be considered regardless of whether it was submitted to leaderboard or not?

Thanks a lot!

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/semeval-2020-task-6-all/75806aab-0395-4d10-af38-b4c06b445490%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/semeval-2020-task-6-all/b8aff464-f5cf-42eb-bb45-edc2b6c32182%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/semeval-2020-task-6-all/cee28036-c579-4588-8d77-515726a9e6d0%40googlegroups.com.

--

Cheers,

Iman