Announcement for score release and paper updates
physionet-challenges
Dear Challengers,
Thank you for another successful Challenge! We had hoped to see everyone this year in Brno, Czech Republic, but we enjoyed sharing CinC with you all the same, and we are hopeful for us all to be together in person next year in Tampere, Finland.
This announcement has important details about your scores on the test data, the final updates to your papers (which you need to implement to be ranked), a final chance for evaluation on the hidden test set, and a survey about your cloud computing needs. Please read it carefully.
Test scores
We have posted preliminary scores and rankings on the test data on https://physionetchallenges.org/2021/leaderboard/
Please note that there are two lists: the official, ranked list of teams sorted by test score, and an unofficial, unranked list of other teams in alphabetical order. The teams in the second list were not ranked because of various failures to adhere to rules, including non-functioning training code, failure to register for CinC or to upload a preprint by the deadline. You may move from list 1 to list 2 if you do not adhere to the instructions below on finalizing your papers and upload by the deadline next week.
Please also note that the "validation" score is the intermediate score that the teams received during the official phase of the Challenge before we ran your final selected code on the hidden test data. Please include the training, validation, and test scores in your papers using the format described in the paper template:
https://physionetchallenges.org/2021/papers/
If you did not receive a final test score, please be clear about that in your paper. Articles that refer to a validation score as if it is the final metric by which to be judged will not be eligible for publication.
Final papers and deadline
Please update your preprints to include the test scores, update your discussion and conclusions, and address any issues with your preprint. Please upload your final papers on Softconf by 23:59 (your local time) on 24 September 2021:
https://www.softconf.com/l/cinc2021/
We’ve already contacted several teams to ask them to fix issues with their preprints, but please don’t rely on us to find these issues. Please read the paper template for instructions, including the following items:
Cite both of the Challenge articles correctly using the references in the CinC template. Specifically they are:
We ask you to cite these articles so we can measure the impact of the Challenge and report this to those that sponsor us. If you fail to cite us, our impact is under-reported and future Challenges are much less likely. We appreciate your help with this.
Do not cite the Challenge websites … and avoid citing websites in general.
Try to avoid citing preprints. They are not peer reviewed and are often low quality.
Cite your other references correctly: https://ieeeauthorcenter.ieee.org/wp-content/uploads/IEEE-Reference-Guide.pdf (we’ve seen many sloppy references where authors’ names are butchered, abbreviations and journal names are are not capitalized, and important information is missing.
Present your results in your abstract and results table like we did in the CinC template for consistency with other teams.
Be clear about your data sets and scores. Include your scores and rankings on the validation and test data. If you did not receive scores on the validation or test data, then say so. Do not describe your “local test set”, which is just confusing. The only test set in the context of the Challenge is the one on which we ran your final code submission.
Do not make misleading or inaccurate statements about your results. In particular, do not claim an inaccurate ranking, or report inaccurate statistics.
Teams that are unable to address these issues by the deadline next week are in danger of having their papers rejected and not being ranked, so please review your papers carefully before you resubmit them!
Focus Issue (and last chance for scoring on the test data)
We invite teams to submit longer descriptions or extensions of their work to the ‘Classification of Multilead ECGs’ focus issue of ‘Physiological Measurement’:
‘Physiological Measurement’ is a journal published by the Institute of Physics, which is a non-profit, has no page charges, provides open access options, and allows you to post personal copies.
To support your submissions to this focus issue, we will score your algorithm *once* on the test set as long as you share a mostly complete draft of your paper before the submission deadline. Please submit by 11 January 2022, and please ask for the focus issue.
Parting Thoughts
We look forward to seeing your revised papers and hope that you will consider submitting extensions of your work to the focus issue. Congratulations to the winners, thank you all for participating, and we hope that you will participate again in the next Challenge!
Best,
Gari, Matt, Nadi, and the rest of the Challenge team.
https://PhysioNetChallenges.org/
Please post questions and comments in the forum. However, if your question reveals information about your entry, then please email challenge at physionet.org. We may post parts of our reply publicly if we feel that all Challengers should benefit from it. We will not answer emails about the Challenge to any other address. This email is maintained by a group. Please do not email us individually.