End of the official phase and next steps to CinC
494 views
Skip to first unread message
PhysioNet Challenge
Aug 24, 2020, 2:55:51 PM8/24/20
to physionet-challenges
Dear Challengers,
The official phase of the challenge is now closed. Thank you to everyone who has submitted entries. We are sorry if we couldn't debug every failed entry in the final moments. As we noted on the Challenge website, we always receive an overwhelming number of entries at the last minute.
We are now in the process of scoring your most recent submissions. Please bear with us. We allow up to 72 hours for training, but the process is not fully automated. Sometimes we can fix small errors, but sometimes it's not obvious — and it's time-consuming to troubleshoot hundreds of unfamiliar complex code bases. Our aim (per the timeline on the Challenge website) is to return these scores to you by the end of this month. Please note that this is not your *final* score. This is just a preliminary test score based on a *subset* of the test data.
You will then have until 2 September 2020 at 23.59 GMT to select which algorithm you wish to have scored on the *full* test set. If you do not select an entry, then we will select the entry that provided the highest score on the subset of the test data (i.e., your highest scoring entry is the default). Please complete this form to select an entry:
This score will be revealed to you after Computing in Cardiology (or at the prize ceremony if you won a prize). You do not need to attend in person to receive the prize, but one of your team members must be registered for the conference and present your work in person or remotely at the appropriate session in the program. Failure to do so will disqualify your team, and this includes failed internet connections or illnesses. For this reason, we strongly suggest that at least two of your team are on hand to present and that you have more than one option for an Internet connection (e.g., phone hotspot and Wi-Fi). Please also note that submitting a preprint on time and your final *updated* paper are critical for prize eligibility.
As noted in an earlier message:
If you have not done so already, then please register for CinC. Your papers and presentations are essential and important (and fun) parts of each Challenge. To be eligible for prizes and the official Challenge rankings, each team must submit a conference paper and give an oral presentation or poster presentation about your work. If you are unable to attend in person, then you can attend remotely this year at a reduced fee. (Please contact the CinC organizers directly for fees, invitation letters, etc.)
Please see the following information (including deadlines) for CinC papers, oral presentations, and poster presentations:
http://www.cinc.org/information-for-computing-in-cardiology-authors/
https://www.cinc2020.org/wp-content/uploads/2020/08/CinC-Speaker-Guidelines_CC_SS.pdf
Like past years, you must submit a preprint of your paper to CinC shortly before the conference (by 6 September 2020) and update it to include your final score and ranking (and any other edits) after the conference (by 30 September 2020). We have prepared a LaTeX template for you to use for your CinC papers (the above website also has a Word template):
https://www.overleaf.com/read/pypkrsjyrqng
Please follow the instructions in this template. In particular, please cite the following reference as the official description of the Challenge. This paper describes the Challenge in general, so you should focus on describing your work:
Please see the following information (including deadlines) for CinC papers, oral presentations, and poster presentations:
http://www.cinc.org/information-for-computing-in-cardiology-authors/
https://www.cinc2020.org/wp-content/uploads/2020/08/CinC-Speaker-Guidelines_CC_SS.pdf
Like past years, you must submit a preprint of your paper to CinC shortly before the conference (by 6 September 2020) and update it to include your final score and ranking (and any other edits) after the conference (by 30 September 2020). We have prepared a LaTeX template for you to use for your CinC papers (the above website also has a Word template):
https://www.overleaf.com/read/pypkrsjyrqng
Please follow the instructions in this template. In particular, please cite the following reference as the official description of the Challenge. This paper describes the Challenge in general, so you should focus on describing your work:
Erick A. Perez Alday, Annie Gu, Amit Shah, Chad Robichaux, An-Kwok Ian Wong, Chengyu Liu, Feifei Liu, Ali Bahrami Rad, Andoni Elola, Salman Seyedi, Qiao Li, Ashish Sharma, Gari D. Clifford, Matthew A. Reyna. Classification of 12-lead ECGs: the PhysioNet/Computing in Cardiology Challenge 2020. Physiol. Meas. Under Review.
Please refer to the timeline (posted on https://physionetchallenges.github.io/2020/#rules-and-deadlines) for clarification on any of this or email the Google Group if you cannot find answers there or in the FAQs.
Best,
The Challenge Organizers
https://physionetchallenges.github.io/
https://physionet.org/
Please post questions and comments in the forum. However, if your question reveals information about your entry, then please email challenge at physionet.org. We may post parts of our reply publicly if we feel that all Challengers should benefit from it. We will not answer emails about the Challenge to any other address. This email is maintained by a group. Please do not email us individually.
https://physionetchallenges.github.io/
https://physionet.org/
Please post questions and comments in the forum. However, if your question reveals information about your entry, then please email challenge at physionet.org. We may post parts of our reply publicly if we feel that all Challengers should benefit from it. We will not answer emails about the Challenge to any other address. This email is maintained by a group. Please do not email us individually.
PhysioNet Challenge
Aug 25, 2020, 10:12:19 PM8/25/20
to physionet-challenges
Dear Challengers,
Some of you recently received error messages from us stating that your entries exceeded our run time and/or memory limits and were therefore unsuccessful. Some of these messages were sent in error, and we will send updated feedback to the affected teams over the next several days. (Some of these messages were not sent in error because some entries did exceed our run time or memory limits, so not everyone who received this error message from us will receive a correction. Please wait to see if you receive a correction before asking if your team was affected.)
We apologize for your justified concern and confusion. We are actively processing your (many) final entries as we speak. While you wait to hear more about these entries, please focus on writing a strong methods section for your preprint. You can check our template and the previous years' papers for instructions and examples:
Best,
Matt
(On behalf of the Challenge team.)
https://physionetchallenges.github.io/
https://physionet.org/
Please post questions and comments in the forum. However, if your question reveals information about your entry, then please email challenge at physionet.org. We may post parts of our reply publicly if we feel that all Challengers should benefit from it. We will not answer emails about the Challenge to any other address. This email is maintained by a group. Please do not email us individually.
PhysioNet Challenge
Aug 27, 2020, 5:25:22 PM8/27/20
to physionet-challenges
Dear Challengers,
We have received many requests over the past several days to correct bugs in your entries, install additional packages on our systems, and accept new or updated submissions. Unfortunately, with the enormous number of entries submitted in the final days of the official phase of the Challenge and the time constraints created by the need to rerun all models on the full test data before CinC, we cannot perform more than the bare minimum of bug fixes or accept additional submissions at this point. We are a small team.
(However, in the event that we need to access your code again because an earlier pull was unsuccessful due to an issue on our end, we strongly recommend that you update your repository with your debugged code, but please do not ask us to make changes for you or to rerun your updated code.)
We understand that many of you have worked for months on the Challenge, and it is frustrating to encounter unexpected issues with your submissions at this point. However, we run the Challenge over several months, and we strongly encourage teams to submit early to both phases of the Challenge to avoid these issues. Unfortunately, many teams who make their first submissions in the days or hours before a deadline will not have successful entries. In these cases, you may not have the opportunity to report your algorithm’s performance on the test data, but please perform (and report) cross validation on the training data whether or not you receive scores on the test data. Be sure to note that these are *training cross-validation* scores, and not *test* scores. If your code was not run on the full test data, then you cannot compare your score directly and you should note this to allow a full and scientific comparison. We will be looking for this and rejecting articles that fail to make this distinction.
Again, we hope to have opportunities to run your code on the test data in the future - please watch out for announcements at the closing ceremony of CinC.
We are looking forward to your papers and presentations at CinC!
All the very best, and thank you again for being a part of the 2020 Challenge!
https://physionetchallenges.github.io/
https://physionet.org/
Please post questions and comments in the forum. However, if your question reveals information about your entry, then please email challenge at physionet.org. We may post parts of our reply publicly if we feel that all Challengers should benefit from it. We will not answer emails about the Challenge to any other address. This email is maintained by a group. Please do not email us individually.
We have received many requests over the past several days to correct bugs in your entries, install additional packages on our systems, and accept new or updated submissions. Unfortunately, with the enormous number of entries submitted in the final days of the official phase of the Challenge and the time constraints created by the need to rerun all models on the full test data before CinC, we cannot perform more than the bare minimum of bug fixes or accept additional submissions at this point. We are a small team.
(However, in the event that we need to access your code again because an earlier pull was unsuccessful due to an issue on our end, we strongly recommend that you update your repository with your debugged code, but please do not ask us to make changes for you or to rerun your updated code.)
We understand that many of you have worked for months on the Challenge, and it is frustrating to encounter unexpected issues with your submissions at this point. However, we run the Challenge over several months, and we strongly encourage teams to submit early to both phases of the Challenge to avoid these issues. Unfortunately, many teams who make their first submissions in the days or hours before a deadline will not have successful entries. In these cases, you may not have the opportunity to report your algorithm’s performance on the test data, but please perform (and report) cross validation on the training data whether or not you receive scores on the test data. Be sure to note that these are *training cross-validation* scores, and not *test* scores. If your code was not run on the full test data, then you cannot compare your score directly and you should note this to allow a full and scientific comparison. We will be looking for this and rejecting articles that fail to make this distinction.
Again, we hope to have opportunities to run your code on the test data in the future - please watch out for announcements at the closing ceremony of CinC.
We are looking forward to your papers and presentations at CinC!
All the very best, and thank you again for being a part of the 2020 Challenge!
Matt, Erick and Gari
https://physionetchallenges.github.io/
https://physionet.org/
Please post questions and comments in the forum. However, if your question reveals information about your entry, then please email challenge at physionet.org. We may post parts of our reply publicly if we feel that all Challengers should benefit from it. We will not answer emails about the Challenge to any other address. This email is maintained by a group. Please do not email us individually.
Reply all
Reply to author
Forward
0 new messages