Final ranking

[Yiwei Jiang]

Hi Song,

The final ranking in the leaderboard is based on the Test Phase results, right? Also, since there are two sub-tasks, i.e. SEEN and UNSEEN, how do you rank participants automatically? Summing the "total" scores from SEEN and UNSEEN sub-tasks?

Best,

Yiwei

[Song Feng]

Hi Yiwei,

Yes, final ranking is based on the TEST phase results for SEEN and UNSEEN respectively. 

Specifically, to quote the high-level ideas explain here ( https://eval.ai/web/challenges/challenge-page/1437/evaluation),

"For final ranking for the awards, we will use the total of the normalized F1-U, SacreBLEU, METEOR and Rouge-L to select the top three teams and then evaluate their results by human annotators to determine the final ranking."

 We would provide more details of human evaluations and how to combine human and automated evaluation results for final ranking later.

Thanks,

Song

--
You received this message because you are subscribed to the Google Groups "dialdoc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dialdoc+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dialdoc/8de5c28d-35ed-47d8-865f-a5bf3e3cdcdbn%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.