Possible error in the new challenge metric

[Bjørn-Jostein Singstad]

Dear Organizers

I have started to work with the data and metrics in the offical phase and I think I might have come across an error (see image)

I would expect the inputs to the murmur challenge metric to be only murmur variables, but there are also outcome labels and classes (see text marked with yellow). In the line marked with green it's more like I would expect, where all variables are related to outcome.

Best regards

Bjørn-Jostein

[PhysioNet Challenge]

Dear Bjørn-Jostein,

Good question! Yes, these variables are correct but confusing. I will add a comment to the code.

For the official phase, we are using two scoring metrics for the two different tasks: a weighted accuracy metric for murmur identification and a cost metric for clinic outcome identification.

However, we can define and compute both scoring metrics for both tasks, but we are only using the clinical outcomes to compute the cost metric for both tasks because the clinical outcome labels are directly related to the treatments and treatment costs for the patients. In this sense, the cost metric effectively evaluates both classifiers' ability to refer patients to a human expert. The clinical outcomes are based on the clinical history, physical examination, analog auscultation, and echocardiogram of each patient, so they are more directly related to whether or not a patient receives treatment than the murmur labels and therefore more directly associated with costs.

Best,
Matt
(On behalf of the Challenge team.)

https://PhysioNetChallenges.org/
https://PhysioNet.org/

Please post questions and comments in the forum. However, if your question reveals information about your entry, then please email info at physionetchallenge.org. We may post parts of our reply publicly if we feel that all Challengers should benefit from it. We will not answer emails about the Challenge to any other address. This email is maintained by a group. Please do not email us individually.

[Bjørn-Jostein Singstad]

Thanks for your answer Matt

Ok. This was very informative and helped me understand the scoring metric much better. I guess other will benefit from reading this too.

Regarding the clinical outcomes; are they from a follow-up study some years later after the recordings?

Best regards

Bjørn-Jostein

[PhysioNet Challenge]

Dear Bjørn-Jostein,

Yes, it was a good and very reasonable question, so thank you for asking it.

https://physionet.org/content/circor-heart-sound/

The clinical outcomes (Abnormal and Normal) were determined during the screening, so you can consider them as contemporaneous with (but different from) the digital auscultations for the murmur labels. Here is the relevant section from the website for the data:

The tag Outcome is the pediatric cardiologist’s overall assessment of the patient’s cardiac condition, as being Normal or Abnormal. This is a holistic diagnosis based on clinical history, physical examination, analog auscultation, echocardiogram, etc. The cardiologists did not have access to the digital auscultations and the murmur gradings when making these diagnoses, and the indicated outcomes do not necessarily imply that the expert had identified a murmur during analog auscultation. The pediatric cardiologists were not the annotators of the digital auscultations.

Best,
Matt
(On behalf of the Challenge team.)

https://PhysioNetChallenges.org/
https://PhysioNet.org/

Please post questions and comments in the forum. However, if your question reveals information about your entry, then please email info at physionetchallenge.org. We may post parts of our reply publicly if we feel that all Challengers should benefit from it. We will not answer emails about the Challenge to any other address. This email is maintained by a group. Please do not email us individually.