PhysioNet Challenge 2024: Scores released
PhysioNet Challenge
Dear Challengers,
Thank you for joining us for a truly interesting and relevant Challenge this year. We were happy to see so many of you this year in Karlsruhe, Germany, and we look forward to seeing many of you next year in São Paulo, Brazil.
This announcement has important information about your scores and papers (which you need to implement to be ranked). Please read it carefully.
Scores
We have posted preliminary scores and rankings on the hidden data:
https://physionetchallenges.org/2024/results/
This page has several tables:
Summary of the Challenge teams, including whether teams were eligible for rankings and prizes
Signal-to-noise ratio (SNR) scores for the ranked teams on the digitization task
SNR scores for the unranked teams on the digitization task
Macro F-measure scores for the ranked teams on the classification task
Macro F-measure scores for the unranked teams on the classification task
All scores in these tables are on the hidden data. The leaderboard scores are for pre-trained models or models trained on a subset of the training set and evaluated on color and black-and-white scans of paper ECGs from the PTB-XL dataset; these are the scores that we reported on the leaderboard during the official phase. The other scores are scores for models retrained on the full test set and evaluated on variants of the paper ECGs from the PTB-XL data, including scans and photographs of clean, stained, and deteriorated ECG papers and photographs of computer monitors showing ECGs.
Please report the scores and ranks from the leaderboard in your papers, and please report other scores as space allows to contextualize the performance of your entries. If we were unable to score your method on the hidden data, then please include cross-validated results on the training data using the Challenge metrics; you can do this even if you have results on the hidden data as well.
Please contact us by Tuesday, 24 September 2024 at 23:59 (your local time) if you believe that your team is on the wrong list, or if you believe that any of the information about your team (except for your team members and title, which we will pull from the conference program) is incorrect. We will update the results afterwards. Please note that your team may move from an official entry to an unofficial entry if you do not adhere to the instructions below on finalizing and uploading your papers by the deadline.
Final papers and final paper deadlines
Please update your four-page conference papers to include the test scores, update your discussion and conclusions, and address any issues with your preprint. Please upload your updated papers on Softconf by 23:59 (your local time) on 27 September 2024:
https://www.softconf.com/n/cinc2024/
You may have noticed that CinC has a final paper upload deadline of 1 October 2024. The Challenge has an earlier deadline to give us time to review your papers and to share feedback with you, and to give you time to upload an updated version of your paper that addresses this feedback before the CinC deadline. Most of our feedback simply addresses items on this checklist, so if you satisfy all of the issues on the checklist, then you will have minimal requests for changes. However, many, if not most, papers need corrections before they can be accepted for publication, so please check this checklist carefully and submit your paper early:
https://physionetchallenges.org/2024/papers/cinc_paper_checklist.pdf
If you did not receive a score on the above result page, then please be clear about that in your paper. Articles that refer to a “local” score as if it were the final metric by which to be judged will not be eligible for publication.
We review each paper and frequently need to ask teams to make corrections that we’ve already requested! Please read the paper template for instructions, including the following items:
Cite the Challenge description and data correctly using the references on this page:
https://physionetchallenges.org/2024/papers/We ask you to cite these articles so that we can measure the impact of the Challenge and report this to those that sponsor us. If you fail to cite this reference article, our impact is under-reported and funding for future Challenges are much less likely. We appreciate your help with this. It also prevents authors from incorrectly describing the data (a problem we often see).
Do not cite the Challenge websites, and avoid citing websites in general.
Cite your other references correctly: https://ieeeauthorcenter.ieee.org/wp-content/uploads/IEEE-Reference-Guide.pdf. We see many sloppy references that have creative versions of names, uncapitalized abbreviations and journal names, and important information (like volume or page numbers) missing. In some cases, you may have entered the information “correctly” in BibTeX, but it compiles incorrectly, so check your PDF!
Present your results in your abstract and results table in the same way we did in the CinC template for consistency with other teams. This makes comparisons easier and ensures you don’t miss key information.
Be clear about your data set. Include your scores and rankings on the hidden - you don't strictly need to provide training or cross validation scores, but if you do, make sure you identify them next to the real hidden data so there’s no misinterpretation. If you did not receive scores on the hidden data, then say so. Do not describe your “local test set”, which is just confusing.
Do not make misleading or inaccurate statements about your results. In particular, do not claim an inaccurate ranking, or report inaccurate statistics. If you are in the unofficial list, the code is not ranked, and you should just say you were not ranked. Do not say where you ‘would’ have been, had you been ranked. This is misleading and confusing.
Again, teams that are unable to address these issues by the deadline are in danger of having their papers rejected and being removed from both the ranked and unofficial unranked lists, so please review your papers carefully before you resubmit them!
Focus Issue and Another Shot at the Hidden Data
We are working to announce a focus issue for the Challenge that will give you another shot at the test set for approaches that build on the many lessons learned from the conference – please stay tuned!
Parting Thoughts
Congratulations to the winners, thank you all for participating in one of the most challenging Challenges to-date, and we hope that you will participate again in the next Challenge!
Best,
The Challenge Team
https://PhysioNetChallenges.org/
Please post questions and comments in the forum. However, if your question reveals information about your entry, then please email challenge at physionet.org. We may post parts of our reply publicly if we feel that all Challengers should benefit from it. We will not answer emails about the Challenge to any other address. This email is maintained by a group. Please do not email us individually.
Felix Krones
Kind regards
Felix
PhysioNet Challenge
(On behalf of the Challenge team)
https://physionetchallenges.org/
Please post questions and comments in the forum. However, if your question reveals information about your entry, then please email in...@physionetchallenges.org. We may post parts of our reply publicly if we feel that all Challengers should benefit from it. We will not answer emails about the Challenge to any other address. This email is maintained by a group. Please do not email us individually.
---
--
You received this message because you are subscribed to the Google Groups "physionet-challenges" group.
To unsubscribe from this group and stop receiving emails from it, send an email to physionet-challe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/physionet-challenges/2fbb25d4-179d-4c42-80f9-8c04b7909123n%40googlegroups.com.
PhysioNet Challenge
This is a quick follow-up to our previous email with two updates and one reminder:
First, we have updated the scores and ranks to include teams that we were able to score but not on all of the variants of the data. Please see this page for the final scores and ranks, and please update your manuscripts as needed:
Again, all scores in these tables are on the hidden data. Your team’s leaderboard score is the highest score received by your team during the official phase; we used this score to rank the teams. We evaluated these entrees on color and black-and-white scans of clean paper ECGs from the PTB-XL dataset. The other scores in the tables are scores for the model that your team chose for scoring after the official phase. We evaluated these models on variants of the paper ECGs from the PTB-XL data, including scans and photographs of clean, stained, and deteriorated ECG papers and photographs of computer monitors showing ECGs.
Second, we have posted a preprint describing the data that we used to evaluate your models. We shared several examples of these data during CinC, and we are planning to publicly share these data to help researchers train more robust and generalizable models for ECG image digitization and classification. Please refer to this paper so that your readers can access the data and a description of the data generation process. The references are here (see the checklist for BibTeX entries):
https://physionetchallenges.org/2024/papers/
Third, please remember to check your paper and upload it on Softconf by 27 September 2024 at 23:59 (your local time) so that we can address common errors before CinC’s final paper deadline:
https://www.softconf.com/n/cinc2024/
Please see the previous announcement for details, but this checklist describes almost everything that you need to check before submitting your paper:
https://physionetchallenges.org/2024/papers/cinc_paper_checklist.pdf
Congratulations again to all of the winners. We hope that many of you will continue to develop your approaches (stay tuned for an announcement), and we hope that all of you will join us again next year.
Best,
The Challenge Team
https://PhysioNetChallenges.org/
https://PhysioNet.org/
Please post questions and comments in the forum. However, if your question reveals information about your entry, then please email challenge at physionet.org. We may post parts of our reply publicly if we feel that all Challengers should benefit from it. We will not answer emails about the Challenge to any other address. This email is maintained by a group. Please do not email us individually.