NeurIPS 2021 MetaDL competition results
3 views
Skip to first unread message
Adrian EL BAZ
Dec 7, 2021, 5:12:17 PM12/7/21
to metalearni...@googlegroups.com
Dear participants,
Evaluation procedure
In order to make it to the final phase, a participant had to perform better than our baseline (ebadrian) which ranked 53.00 during the feedback phase. Any teams that didn't out-perform this baseline were disqualified for the final phase.
Then, each submission was run 3 times. Each run corresponds to a different ingestion seed while the scoring seed remains the same. Hence, this essentially has an impact on how the meta-train data is fed to your algorithms.
The last step to obtain the final rank of participants is to rank submissions for each meta-dataset and to average these ranks, as we did in the feedback phase.
As you can see, the results were close. Indeed, there is a tie between the 2nd and 3rd best teams. In this situation, our competition rules state that the algorithm that was submitted before the other one is considered "better". Therefore, the top 3 rankings is the following:
A final note: ForeverYong's submission couldn't run on the 5th meta-dataset without a small change in their configuration. We've decided to assign the worst rank on the 5th meta-dataset to their submission. For completeness we evaluated the modified version (one parameter in config.gin) of their submission on this meta-dataset, corresponding to the red cell in the table: 0,93900 which ranked 1st.
The ForeverYong team will be discussing their solution in a presentation during the workshop tomorrow. You can find the full program of our workshop here.
Congratulations to the winners and thank you again for participating !
NB: If you have questions/comments about the evaluation protocol, please ask them before the workshop begins. Our apologies for the short notice.
Kind regards,
The organizing team
First of all, thank you all for your participation in the 2nd edition of the MetaDL competition. I've attached a screenshot of the competition results that I will officially announce tomorrow during our dedicated workshop. Here are some details about how these results were calculated.
Evaluation procedure
In order to make it to the final phase, a participant had to perform better than our baseline (ebadrian) which ranked 53.00 during the feedback phase. Any teams that didn't out-perform this baseline were disqualified for the final phase.
Then, each submission was run 3 times. Each run corresponds to a different ingestion seed while the scoring seed remains the same. Hence, this essentially has an impact on how the meta-train data is fed to your algorithms.
For each submission and for each meta-dataset, we took the minimum performance among the 3 different runs. You can refer to the final results in the performance table attached to this email.
The last step to obtain the final rank of participants is to rank submissions for each meta-dataset and to average these ranks, as we did in the feedback phase.
As you can see, the results were close. Indeed, there is a tie between the 2nd and 3rd best teams. In this situation, our competition rules state that the algorithm that was submitted before the other one is considered "better". Therefore, the top 3 rankings is the following:
- 1st prize: ForeverYong
- 2nd prize: henrygouk (1st of October)
- 3rd prize: padawan (2nd of October)
A final note: ForeverYong's submission couldn't run on the 5th meta-dataset without a small change in their configuration. We've decided to assign the worst rank on the 5th meta-dataset to their submission. For completeness we evaluated the modified version (one parameter in config.gin) of their submission on this meta-dataset, corresponding to the red cell in the table: 0,93900 which ranked 1st.
The ForeverYong team will be discussing their solution in a presentation during the workshop tomorrow. You can find the full program of our workshop here.
Congratulations to the winners and thank you again for participating !
NB: If you have questions/comments about the evaluation protocol, please ask them before the workshop begins. Our apologies for the short notice.
Kind regards,
The organizing team
Reply all
Reply to author
Forward
0 new messages