2023 PhysioNet Challenge scores released
530 views
Skip to first unread message
PhysioNet Challenge
Oct 10, 2023, 10:57:51 PM10/10/23
to physionet-challenges
Dear Challengers,
Thank you for another successful Challenge! We had the privilege to host you at CinC 2023 in Atlanta, Georgia, USA, and we look forward to seeing you in Karlruhe next year!
This announcement has important information about the scores on the test data and the updates to your papers (which you need to implement to be ranked). Please read it carefully.
Test scores
This page has several tables:
Please contact us by Friday, 13 October 2022 at 23:59 (your local time) if you believe that your team is on the wrong list, or if you believe that any of the information about your team is incorrect. We will update the results afterwards. Please note that your team may move from an official entry to an unofficial entry if you do not adhere to the instructions below on finalizing and uploading your papers by the deadline.
Final papers and final paper deadlines
Please update your four-page conference papers to include the test scores, update your discussion and conclusions, and address any issues with your preprint. Please upload your final papers on Softconf by 23:59 (your local time) on 15 October 2023:
https://www.softconf.com/m/cinc2023/
You may have noticed that CinC has a final paper upload deadline of 20 October 2023. The Challenge has an earlier deadline to give us time to review your papers and to share feedback with you, and to give you time to upload an updated version of your paper that addresses this feedback before the CinC deadline. Most of our feedback simply addresses items on this checklist, so if you satisfy all of the issues on the checklist, then we will have minimal feedback. However, many, if not most, papers need corrections before they can be accepted for publication, so please check this checklist carefully and submit your paper early:
https://physionetchallenges.org/2023/papers/cinc_paper_checklist.pdf
In particular, the above tables include the scores for your chosen models on the training, validation, and test sets. Please note that the "validation" score is the intermediate score that the teams received during the official phase of the Challenge before we ran your final selected code on the test set. Please include the training, validation, and test scores, and, if ranked, your rank on the test set, in your papers using the format described in the paper template:
https://physionetchallenges.org/2023/papers/
If you did not receive a final test score, please be clear about that in your paper. Articles that refer to a validation score or “local” test score as if it were the final metric by which to be judged will not be eligible for publication.
We review each paper and frequently need to ask teams to make corrections that we’ve already requested! Please read the paper template for instructions, including the following items:
Focus Issue and Another Shot at the Test Set
We are working to announce a focus issue for the Challenge that will give you another shot at the test set for approaches that build on the many lessons learned from the conference – please stay tuned!
Parting Thoughts
Congratulations to the winners, thank you all for participating in one of the most challenging Challenges to-date, and we hope that you will participate again in the next Challenge!
Best,
The Challenge team
https://PhysioNetChallenges.org/
https://PhysioNet.org/
Thank you for another successful Challenge! We had the privilege to host you at CinC 2023 in Atlanta, Georgia, USA, and we look forward to seeing you in Karlruhe next year!
This announcement has important information about the scores on the test data and the updates to your papers (which you need to implement to be ranked). Please read it carefully.
Test scores
We have posted preliminary scores and rankings on the test data: https://physionetchallenges.org/2023/results/
This page has several tables:
- A table with a summary of the Challenge teams
- Four tables (at 12, 24, 48, and 72 hours after return of spontaneous circulation) with scores and rankings for official entries
- Four tables (again at 12, 24, 48, and 72 hours after return of spontaneous circulation) with scores for unofficial entries
Please contact us by Friday, 13 October 2022 at 23:59 (your local time) if you believe that your team is on the wrong list, or if you believe that any of the information about your team is incorrect. We will update the results afterwards. Please note that your team may move from an official entry to an unofficial entry if you do not adhere to the instructions below on finalizing and uploading your papers by the deadline.
Final papers and final paper deadlines
Please update your four-page conference papers to include the test scores, update your discussion and conclusions, and address any issues with your preprint. Please upload your final papers on Softconf by 23:59 (your local time) on 15 October 2023:
https://www.softconf.com/m/cinc2023/
You may have noticed that CinC has a final paper upload deadline of 20 October 2023. The Challenge has an earlier deadline to give us time to review your papers and to share feedback with you, and to give you time to upload an updated version of your paper that addresses this feedback before the CinC deadline. Most of our feedback simply addresses items on this checklist, so if you satisfy all of the issues on the checklist, then we will have minimal feedback. However, many, if not most, papers need corrections before they can be accepted for publication, so please check this checklist carefully and submit your paper early:
https://physionetchallenges.org/2023/papers/cinc_paper_checklist.pdf
In particular, the above tables include the scores for your chosen models on the training, validation, and test sets. Please note that the "validation" score is the intermediate score that the teams received during the official phase of the Challenge before we ran your final selected code on the test set. Please include the training, validation, and test scores, and, if ranked, your rank on the test set, in your papers using the format described in the paper template:
https://physionetchallenges.org/2023/papers/
If you did not receive a final test score, please be clear about that in your paper. Articles that refer to a validation score or “local” test score as if it were the final metric by which to be judged will not be eligible for publication.
We review each paper and frequently need to ask teams to make corrections that we’ve already requested! Please read the paper template for instructions, including the following items:
- Cite the Challenge description and data correctly using the references on this page: https://physionetchallenges.org/2023/papers/
- We ask you to cite these articles so that we can measure the impact of the Challenge and report this to those that sponsor us. If you fail to cite this reference article, our impact is under-reported and funding for future Challenges are much less likely. We appreciate your help with this. It also prevents authors from incorrectly describing the data (a problem we often see).
- Do not cite the Challenge websites, and avoid citing websites in general.
- Cite your other references correctly: https://ieeeauthorcenter.ieee.org/wp-content/uploads/IEEE-Reference-Guide.pdf. We see many sloppy references that have creative versions of names, uncapitalized abbreviations and journal names, and important information (like volume or page numbers) missing. In some cases, you may have entered the information “correctly” in BibTeX, but it compiles incorrectly, so check your PDF!
- Present your results in your abstract and results table in the same way we did in the CinC template for consistency with other teams. This makes comparisons easier and ensures you don’t miss key information.
- Be clear about your data sets (train, cross-validation on training data, validation, test) and metrics/scores. Include your scores and rankings on the validation and test data - you don't strictly need to provide training or cross validation scores, but if you do, make sure you identify them next to the real test data so there’s no misinterpretation. If you did not receive scores on the validation or test data, then say so. Do not describe your “local test set”, which is just confusing. The only test set in the context of the Challenge is the one on which we ran your final code submission.
- Do not make misleading or inaccurate statements about your results. In particular, do not claim an inaccurate ranking, or report inaccurate statistics. If you are in the unofficial list, the code is not ranked, and you should just say you were not ranked. Do not say where you ‘would’ have been, had you been ranked. This is misleading and confusing.
Focus Issue and Another Shot at the Test Set
We are working to announce a focus issue for the Challenge that will give you another shot at the test set for approaches that build on the many lessons learned from the conference – please stay tuned!
Parting Thoughts
Congratulations to the winners, thank you all for participating in one of the most challenging Challenges to-date, and we hope that you will participate again in the next Challenge!
Best,
The Challenge team
https://PhysioNetChallenges.org/
https://PhysioNet.org/
Please post questions and comments in the forum. However, if your question reveals information about your entry, then please email challenge at physionet.org. We may post parts of our reply publicly if we feel that all Challengers should benefit from it. We will not answer emails about the Challenge to any other address. This email is maintained by a group. Please do not email us individually.
Reply all
Reply to author
Forward
0 new messages