Official phase of the PhysioNet Challenge begins!

685 views
Skip to first unread message

PhysioNet Challenge

unread,
May 1, 2021, 11:41:10 PM5/1/21
to physionet-challenges
Dear Challengers,

We are pleased to announce the start of the official phase of the PhysioNet/Computing in Cardiology Challenge 2021: “Will Two Do? Varying Dimensions in Electrocardiography”.

During the unofficial phase, we were heartfelt to receive some very useful public and private comments, and offers of help, which have helped improve the Challenge.

The official phase of this year’s Challenge brings many additions, including new lead combinations, an additional training set with approximately 45,000 twelve-lead ECGs (most of which have never been released before), new test data, updated MATLAB and Python code, and a slightly updated scoring function with additional classes. Due to these many changes, we expect to make several updates or clarifications over the next few days.

New lead combinations
To better understand the differential utility of reduced-lead ECGs, we have changed our two-lead ECGs and added four-lead ECGs.

Twelve leads: I, II, III, aVR, aVL, aVF, V1, V2, V3, V4, V5, V6
Six leads: I, II, III, aVR, aVL, aVF
Four leads: I, II, III, V2
Three leads: I, II, V2
Two leads: I, II

Please see the Objective section of the Challenge website for more details.

Thank you to Andrew Walsh, Kim Barnett, Joel Xue and Dave Albert for useful input on this issue.

Expanded training set
Jianwei Zheng and colleagues generously shared approximately 45,000 twelve-lead ECG recordings from Shaoxing University. A description of these new data can be found in Zheng et al. [1, 2].

With their support, we have added these recordings as an additional training set for this year’s Challenge. These recordings more than double the size of our training set to over 88,000 recordings.

Please see the Data Sources section of the Challenge website for more details.

Updated MATLAB and Python code
We have updated our example training and test code in both MATLAB and Python to address your feedback and our observations from the unofficial phase. Most of these changes are minor, but due to the change in lead combinations, we recommend that you download the updated code and test your models using it before submitting your code again.

Please see the Algorithms section of the Challenge website for more details, including the MATLAB and Python code repositories.

Updated scoring
We are updating our scoring function to incorporate the new training data and lead combinations. We plan to release it in the next several days, and we plan to start scoring your submissions again afterwards.

Please see the Scoring section of the website for more details, including the scoring code repository.

CinC abstracts
Your abstracts are under review, and we expect for you to receive decisions on your abstracts by 1 June 2021. If you do not receive a decision on this date, then please wait or check the CinC 2021 webpage for updates -- the abstract submission deadline extension may push this date back.

CinC 2021
Like last year’s conference, CinC 2021 will be a hybrid conference with in-person and remote attendance options. Both in-person and remote attendees are eligible to compete in the Challenge, and we look forward to a strong official phase.

Best,
Gari, Matt, Nadi, Erick, and the rest of the Challenge team

https://PhysioNetChallenges.org/
https://PhysioNet.org/

Please post questions and comments in the forum. However, if your question reveals information about your entry, then please email challenge at physionet.org. We may post parts of our reply publicly if we feel that all Challengers should benefit from it. We will not answer emails about the Challenge to any other address. This email is maintained by a group. Please do not email us individually.

References
[1] Zheng, J., Zhang, J., Danioko, S. et al. A 12-lead electrocardiogram database for arrhythmia research covering more than 10,000 patients. Sci Data 7, 48 (2020). https://doi.org/10.1038/s41597-020-0386-x
(This describes the first 10,000 ECGs in the new data. )

[2] Zheng, J., Chu, H., Struppa, D. et al. Optimal Multi-Stage Arrhythmia Classification Approach. Sci Rep 10, 2898 (2020). https://doi.org/10.1038/s41598-020-59821-7
(This describes an analysis of the data in [1] by the same authors.)

wenh06

unread,
May 3, 2021, 3:31:17 PM5/3/21
to physionet-challenges
Should the dx_mapping csv files be updated? Records from the new tranche have Dx that are not in these files (e.g. "JS00011" has  "#Dx: 426177001,55827005", with "55827005" not listed in the csv files").

PhysioNet Challenge

unread,
May 3, 2021, 3:34:50 PM5/3/21
to physionet-challenges
Dear Wen Hao,

Yes, we will update the diagnosis mappings in the scoring repository to include new diagnoses from the new dataset, and we will update the scoring code for the new diagnoses as well. Please expect these updates within days.

Best,
Matt
(On behalf of the Challenge team)


https://PhysioNetChallenges.org/
https://PhysioNet.org/

Please post questions and comments in the forum. However, if your question reveals information about your entry, then please email challenge at physionet.org. We may post parts of our reply publicly if we feel that all Challengers should benefit from it. We will not answer emails about the Challenge to any other address. This email is maintained by a group. Please do not email us individually.

Pierre-G Aublin

unread,
May 6, 2021, 10:39:51 PM5/6/21
to physionet-challenges
Dear Organizers,
Regarding the doubling of the training set size, is the training time limit maintained to 36 hours (gpu) or will there be an extra ?
Yours faithfully
Aublin Pierre.

physionet-challenges

unread,
May 6, 2021, 10:48:09 PM5/6/21
to physionet-challenges
Dear Aublin,

The time limit of training code using a GPU was increased to 48 hours and the training time limit without a GPU remained at 72 hours.

For future updates and more information about the computational resources, please visit the current Challenge FAQ:
https://physionetchallenges.org/2021/faq/#computational

Best,

Nadi
(On behalf of the Challenge team)


philip....@gmail.com

unread,
May 11, 2021, 2:51:25 AM5/11/21
to physionet-challenges
Dear organizers,

Just curious, changing the 2-lead configuration to I,II makes it a strict subset of the other configurations.

The 3-lead and new 4-lead configuration includes V2 which is not included in the 6-lead configuration.

If rationale for changing the 2-lead to a strict subset was to better compare with the others (and I'm not sure that was your only motivation), why not do the same for 3- and 4-lead?

Thanks,
Philip

physionet-challenges

unread,
May 11, 2021, 2:57:33 AM5/11/21
to physionet-challenges
Dear Philip,

Thanks for your great question. We were hoping to receive more questions about the combinations of leads during the unofficial phase.

The aim of this Challenge is to compare different combinations and subsets of leads and it is indeed the question which you are asked to address during the Challenge:

Which combinations or subsets of leads are comparable? Which leads carry extra information and why do doctors bother to measure these "redundant" leads? Which leads are really a separate lead and which are really derived from the other leads? 
If we see no difference in the performance, then we will know for sure that there is no additional information. 

There are also many factors contributing to the problem, e.g sometimes, leads are noisy, inverted or missing, we are not sure which devices recorded ECG signals in the datasets or exactly how leads were connected, etc. These make it a more challenging question to answer.

We certainly like to hear your thoughts on this.

Best,

Nadi
(On behalf of the Challenge team)

PhysioNet Challenge

unread,
May 17, 2021, 12:03:12 PM5/17/21
to physionet-challenges
Dear Challengers,

We plan to start accepting submissions for the official phase of this year’s Challenge by the end of this week.

Thanks for your continued questions, comments, and interest, all of which have allowed us to substantially improve the Challenge from the unofficial phase.

We will make an announcement that we have reopened scoring after making a few final changes for the official phase.

Best,
Matt
(On behalf of the Challenge team.)


https://PhysioNetChallenges.org/
https://PhysioNet.org/

Please post questions and comments in the forum. However, if your question reveals information about your entry, then please email challenge at physionet.org. We may post parts of our reply publicly if we feel that all Challengers should benefit from it. We will not answer emails about the Challenge to any other address. This email is maintained by a group. Please do not email us individually.

On Saturday, May 1, 2021 at 11:41:10 PM UTC-4 PhysioNet Challenge wrote:

Karen Liu

unread,
May 18, 2021, 6:59:55 AM5/18/21
to physionet-challenges
Dear Organizers, 
I want to confirm that 48 hours is for training one model or five models?
Yours faithfully,
Karen Liu

PhysioNet Challenge

unread,
May 18, 2021, 7:03:35 AM5/18/21
to physionet-challenges
Dear Karen,

Yes, the total training time limit (currently 48 hours with a GPU and 72 hours without a GPU) is for training all five models. We impose few limits on how you train your models beyond the technical limitations, so you are welcome to reuse parts of your training code for your different models.

Best,
Matt

(On behalf of the Challenge team.)


https://PhysioNetChallenges.org/
https://PhysioNet.org/

Please post questions and comments in the forum. However, if your question reveals information about your entry, then please email challenge at physionet.org. We may post parts of our reply publicly if we feel that all Challengers should benefit from it. We will not answer emails about the Challenge to any other address. This email is maintained by a group. Please do not email us individually.

PhysioNet Challenge

unread,
May 23, 2021, 9:51:19 PM5/23/21
to physionet-challenges
Dear Challengers,

We are pleased to announce that we are now accepting submissions for official phase of this year's Challenge! Please see the Challenge webpage, the submission instructions, or this form to submit your official phase entries.

Please see our earlier announcement for more details about changes from the unofficial phase of the Challenge. In short, we have more than doubled the size of the training set, changed the reduced lead combinations, updated our example MATLAB and Python classifier code, and updated the scoring function for the new classes from the new training databases. In particular, we updated the headers for the new training databases and the lists of scored and unscored diagnoses since our last announcement a week ago.

Thank you again for your feedback and offers of help, which have allowed us to make these changes. We hope that they will help you with your models, and we look forward to your submissions.

Best,
Matt
(On behalf of the Challenge team.)

P.S.: You should receive your abstract decisions in the next week or two -- stay tuned!

https://PhysioNetChallenges.org/
https://PhysioNet.org/

Please post questions and comments in the forum. However, if your question reveals information about your entry, then please email challenge at physionet.org. We may post parts of our reply publicly if we feel that all Challengers should benefit from it. We will not answer emails about the Challenge to any other address. This email is maintained by a group. Please do not email us individually.

Reply all
Reply to author
Forward
0 new messages