We have received many requests over the past several days to correct bugs in your entries, install additional packages on our systems, and accept new or updated submissions. Unfortunately, with the enormous number of entries submitted in the final days of the official phase of the Challenge and the time constraints created by the need to rerun all models on the full test data before CinC, we cannot perform more than the bare minimum of bug fixes or accept additional submissions at this point. We are a small team.
(However, in the event that we need to access your code again because an earlier pull was unsuccessful due to an issue on our end, we strongly recommend that you update your repository with your debugged code, but please do not ask us to make changes for you or to rerun your updated code.)
We understand that many of you have worked for months on the Challenge, and it is frustrating to encounter unexpected issues with your submissions at this point. However, we run the Challenge over several months, and we strongly encourage teams to submit early to both phases of the Challenge to avoid these issues. Unfortunately, many teams who make their first submissions in the days or hours before a deadline will not have successful entries. In these cases, you may not have the opportunity to report your algorithm’s performance on the test data, but please perform (and report) cross validation on the training data whether or not you receive scores on the test data. Be sure to note that these are *training cross-validation* scores, and not *test* scores. If your code was not run on the full test data, then you cannot compare your score directly and you should note this to allow a full and scientific comparison. We will be looking for this and rejecting articles that fail to make this distinction.
Again, we hope to have opportunities to run your code on the test data in the future - please watch out for announcements at the closing ceremony of CinC.
We are looking forward to your papers and presentations at CinC!
All the very best, and thank you again for being a part of the 2020 Challenge!
Matt, Erick and Gari
Please post questions and comments in the forum. However, if your question reveals information about your entry, then please email challenge at physionet.org
. We may post parts of our reply publicly if we feel that all Challengers should benefit from it. We will not answer emails about the Challenge to any other address. This email is maintained by a group. Please do not email us individually.