properties of the scoring function

[xnej...@gmail.com]

Dear challenge organizers,

We have found interesting properties of the scoring function. 

Given an example dataset, where all recordings are normal class and correctly classified by a classifier. The resulting challenge score is equal to 0, however, 100% of the recording are correctly classified. 

I would suggest changing score normalization in a way that the inactive classifier (outputs always 0.0 probability) has the challenge score equal to 0. 

Your current definition assumes that inactive classifier outputs always normal class. However, normal class is not mutually exclusive to other classes. 

Can you please comment on why normal class normalization is used? 

Thank you,

Petr Nejedly

[PhysioNet Challenge]

Dear Petr,

Thank you for your observation and suggestions.

We wanted to normalize the scoring function so that classifiers can receive either a positive or a negative score depending on their performance.

For simplicity, we chose to map the score for a perfect classifier to 1 and the score for a classifier that only returns the sinus rhythm class to 0. You are certainly correct that this normalization does not work if there are no cardiac abnormalities in the recordings, but that is not a problem for our data. "Random" classifiers, and classifiers that are not well calibrated, tend to receive negative scores from this scoring function.

If we were to map a completely inactive classifier to 0, then classifiers could not receive a negative score. We also considered other normalizations but did not find them to be more interpretable. Also, the choice of normalization does not ultimately affect the ranking of the classifiers.

Best,

Matt

(On behalf of the Challenge team.)

https://PhysioNetChallenges.org/
https://PhysioNet.org/

Please post questions and comments in the forum. However, if your question reveals information about your entry, then please email challenge at physionet.org. We may post parts of our reply publicly if we feel that all Challengers should benefit from it. We will not answer emails about the Challenge to any other address. This email is maintained by a group. Please do not email us individually.