Final Prediction File Upload Opens May 30th
145 views
Skip to first unread message
Nick Landia
May 19, 2022, 1:42:12 PM5/19/22
to RecSys-Challenge-2022
This just went up on the leaderboard website reposting it here:
On May 30th 7am PST we go into the Testing phase of the challenge.
The teams will be fixed from this point onwards. You won’t be able to merge teams or make new accounts.
Submission of the final prediction files will open. You can upload multiple times but only the last successfully evaluated submission will count. When submitting you will receive feedback on whether the file was evaluated successfully, but the score and rank will not be disclosed until the winner announcement.
Before we allow you to make a final prediction file submission for your team the organiser of the team has to fill in the author info for the team. The author info will include full name, affiliation and email address of all team members and has to match exactly the author list of your paper submission. Once entered this will not be editable any more, please make sure the entered information is complete before you click submit. Each author can only be in one team. If there is someone who is not signed up to your team but should be on the paper later (e.g. your supervisor or adviser) make sure to include them in the author info for your team. Adding people to the author list later will not be allowed. This is a measure to prevent multiple submissions for the same team under different accounts and everyone hopping on the author list of the winning version later.
You will have time to upload your final prediction file until the challenge ends on 2022-06-14 23:59:59 PST. We recommend doing your first upload of the final predictions as early as possible in case there are format errors during evaluation. You can always upload revisions of your final predictions afterwards.
Benedikt S
May 23, 2022, 11:38:10 AM5/23/22
to RecSys-Challenge-2022
Hello Nick,
thanks for sharing the instructions for the final phase.
Do we only need to submit the final predictions? Do we not submit the code and/or make it publicly available?
How will it be ensured that a team does not accidental use the test data for training/feature engineering?
Best
Benedikt
Benedikt S
May 25, 2022, 1:37:33 PM5/25/22
to RecSys-Challenge-2022
Hello Nick,
will it be still possible to submit to public leaderboard and see scores after Math 30th 7am until 2022-06-04?
Best
Benedikt
On Thursday, May 19, 2022 at 7:42:12 PM UTC+2 Nick Landia wrote:
Nick Landia
May 25, 2022, 2:30:51 PM5/25/22
to RecSys-Challenge-2022
Hi Benedikt,
- Do we only need to submit the final predictions? Do we not submit the code and/or make it publicly available?
You only need to submit the final predictions in the submission system itself but the winning teams will need to submit the code as well along with the paper submission. Instructions on code submission will follow.
- How will it be ensured that a team does not accidental use the test data for training/feature engineering?
This is the same as any other paper submission, your code has to execute the experiment the way you have described in the paper. The approach and the code have to conform to the rules. Ensuring this is the responsibility of each author. One easy way to do this is to ensure your code does not read the test file until it absolutely needs to, instead of reading in everything at the start and then accidentally including the test sessions in model training only read the test sessions at the point when you query the already trained model for predictions. It can still go wrong of course, but the responsibility of producing high quality, accurate experiment runs is with the authors. If data is used that is not allowed by the rules that would make the result and rank of the team invalid.
- will it be still possible to submit to public leaderboard and see scores after Math 30th 7am until 2022-06-04?
yes, you will still be able to evaluate the leaderboard test set predictions until the challenge ends (2022-06-14).
You only need to submit the final predictions in the submission system itself but the winning teams will need to submit the code as well along with the paper submission. Instructions on code submission will follow.
- How will it be ensured that a team does not accidental use the test data for training/feature engineering?
This is the same as any other paper submission, your code has to execute the experiment the way you have described in the paper. The approach and the code have to conform to the rules. Ensuring this is the responsibility of each author. One easy way to do this is to ensure your code does not read the test file until it absolutely needs to, instead of reading in everything at the start and then accidentally including the test sessions in model training only read the test sessions at the point when you query the already trained model for predictions. It can still go wrong of course, but the responsibility of producing high quality, accurate experiment runs is with the authors. If data is used that is not allowed by the rules that would make the result and rank of the team invalid.
- will it be still possible to submit to public leaderboard and see scores after Math 30th 7am until 2022-06-04?
yes, you will still be able to evaluate the leaderboard test set predictions until the challenge ends (2022-06-14).
Best,
Nick
Reply all
Reply to author
Forward
0 new messages