Official Results
51 views
Skip to first unread message
Stephen Mayhew
Apr 16, 2020, 1:19:16 PM4/16/20
to duolingo-sharedtask-2020
Hello all, We have published the official results in the table below, and on the shared task page! See also this spreadsheet for more details. Congratulations to all the teams, and thanks for participating! Once again, a reminder that system description papers are due on Monday (midnight UTC). Paper submission and reviews will be handled via SoftConf (more details on the shared task page). As you work on it, keep in mind that we are interested not only in top-performing systems (i.e., metrics), but also meaningful findings (i.e., insights for language and/or learning)! Please also include a discussion of any additional corpora used.
Stephen user | hu | ja | ko | pt | vi |
|---|---|---|---|---|---|
| jbrem | 0.555 | 0.318 | 0.404 | 0.552 | 0.558 |
| Nickeilf | -- | -- | -- | 0.551 | -- |
| rakchada | 0.552 | -- | -- | 0.544 | -- |
| jspak3 | -- | -- | 0.312 | -- | -- |
| sweagraw | 0.469 | 0.294 | 0.255 | 0.525 | 0.539 |
| Masahiro | -- | 0.283 | -- | -- | -- |
| mzy | -- | 0.260 | -- | -- | -- |
| hzguo | -- | 0.239 | -- | -- | -- |
| dcu | -- | -- | -- | 0.460 | -- |
| jindra.helcl | 0.435 | 0.213 | 0.206 | 0.412 | 0.377 |
| darkside | -- | 0.194 | -- | -- | -- |
| NAGOUDI | -- | -- | -- | 0.376 | -- |
| Anush | -- | -- | -- | 0.305 | -- |
| STAPLE_aws_baseline | 0.281 | 0.043 | 0.041 | 0.213 | 0.198 |
| STAPLE_fairseq_baseline | 0.124 | 0.033 | 0.049* | 0.136 | 0.254* |
Stephen Mayhew
Apr 16, 2020, 1:31:51 PM4/16/20
to duolingo-sharedtask-2020
Oops, the spreadsheet wasn't accessible. The permissions should be fixed now. Thanks to those who pointed it out!
Stephen
Matt Post
Apr 16, 2020, 1:45:56 PM4/16/20
to Stephen Mayhew, duolingo-sharedtask-2020
Thanks, Stephen. With this change in reporting, we’re not sure which of our submissions at the end was the best. Could you add some more information to the spreadsheet? What would help:
1. Adding the scores of *all* submissions per language
2. Adding a column with the date+time of each submission
Probably #2 would be sufficient though I’d also be interested in seeing #1.
matt
--
You received this message because you are subscribed to the Google Groups "duolingo-sharedtask-2020" group.
To unsubscribe from this group and stop receiving emails from it, send an email to duolingo-sharedtas...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/duolingo-sharedtask-2020/5028113b-fd26-4f8e-93f9-cbe34e698aeb%40googlegroups.com.
Stephen Mayhew
Apr 16, 2020, 2:12:13 PM4/16/20
to Matt Post, duolingo-sharedtask-2020
Ah, there's an ID attached to each submission in the spreadsheet, but I've just found that CodaLab doesn't give you your submission ID. Thanks for pointing that out.
I've added 4 tabs to the spreadsheet: Test Metadata, Test Submissions, Dev Metadata, Dev Submissions. It's not presented in the most beautiful way, but I hope the information is helpful.
Stephen
Zhenhao Li
May 11, 2020, 12:02:42 PM5/11/20
to duolingo-sharedtask-2020
Hello Stephen,
Thanks for organizing the shared task. I know it might be a bit late to post a question about the official result, but just out of curiosity, the official results are different from the leaderboard result shown on Codalab. Is there a minor change in the calculation for the final score?
Best,
Zhenhao
Stephen Mayhew
May 11, 2020, 12:31:02 PM5/11/20
to Zhenhao Li, duolingo-sharedtask-2020
Hi Zhenhao,
No problem! This is because of a slightly odd way that CodaLab reports scores. We made a note on the webpage -- I'll quote it here:
These results differ slightly from the CodaLab leaderboard. CodaLab chooses to display results from a team's entire submission (according to some comparison function). We have chosen to select each team's highest performing score for each language, which are not necessarily all from the same submission.
Stephen
--
You received this message because you are subscribed to the Google Groups "duolingo-sharedtask-2020" group.
To unsubscribe from this group and stop receiving emails from it, send an email to duolingo-sharedtas...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/duolingo-sharedtask-2020/34136829-abc4-4dab-8e9e-a47a9df420ad%40googlegroups.com.
Zhenhao Li
May 12, 2020, 7:10:59 AM5/12/20
to duolingo-sharedtask-2020
Hi Stephen,
Thanks for answering! I still find that there is a minor mismatch on the precision score on the test submission between the Google spreadsheet and the Codelab.
For example in the two pics below (Test Submission), the recall score is the same so I think the two correspond to the same submission. The precision score is 0.7048 for Codalab leaderboard, but 0.7053 in the spreadsheet.
Best,
Zhenhao
To unsubscribe from this group and stop receiving emails from it, send an email to duolingo-sharedtask-2020+unsub...@googlegroups.com.
Stephen Mayhew
May 18, 2020, 5:09:13 PM5/18/20
to Zhenhao Li, duolingo-sharedtask-2020
Hi Zhenhao,
Thanks for pointing this out. We've been looking into it, and you're right it's not the "best submission from CodaLab" explanation. When we downloaded all the submissions, we noticed a minor bug in the scoring script that sometimes failed to count the last prompt in a submission. We fixed it, and it made a small difference in a few submissions. This is one place it made a difference. Also, notice that the recall values you highlight between the spreadsheet and CodaLab aren't identical, because with rounding (which CodaLab does) the value would be 0.5174.
That said, we've done statistical testing on consecutive teams in each leaderboard (see the spreadsheet for details), and this particular difference, as you might expect, is not significant.
Stephen
To unsubscribe from this group and stop receiving emails from it, send an email to duolingo-sharedtas...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/duolingo-sharedtask-2020/34136829-abc4-4dab-8e9e-a47a9df420ad%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "duolingo-sharedtask-2020" group.
To unsubscribe from this group and stop receiving emails from it, send an email to duolingo-sharedtas...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/duolingo-sharedtask-2020/5e32c0f5-3ae8-4054-92a1-49e0954997b1%40googlegroups.com.
Zhenhao Li
May 19, 2020, 6:45:08 AM5/19/20
to duolingo-sharedtask-2020
Hi Stephen,
Thanks for the answer, and again thanks for organizing this great shared task!
Cheers,
Zhenhao
To unsubscribe from this group and stop receiving emails from it, send an email to duolingo-sharedtask-2020+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/duolingo-sharedtask-2020/34136829-abc4-4dab-8e9e-a47a9df420ad%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "duolingo-sharedtask-2020" group.
To unsubscribe from this group and stop receiving emails from it, send an email to duolingo-sharedtask-2020+unsub...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages