I want to challenge the challenge metric of CinC2021/2020

[wenh06]

Recently I noticed that the challenge metric of CinC2021/2020 might have problems. The scoring weight matrix has a minimum value of 0.15, and a mean value (excluding the diagonal ones) of  0.38. This makes it possible that a randomly generated output can receive a challenge score that seems to be acceptable. An example is given in this notebook.

I think the above example explains the weird phenomenon in the official results spreadsheet that some teams received very low (even zero) accuracy. The accuracy is computed for strict matches. The challenge metric punishes models that always choose the sinus rhythm class. Should it also punish models that seldom predict correctly?

[PhysioNet Challenge]

Dear Wen Hao,

Thanks for this feedback and your example.

As you noticed, the scoring metric for the 2020 and 2021 Challenges assigns different weightings to different misclassification errors. With the weights that we defined for these Challenges, a classifier may receive a higher reward for identifying a similar cardiac abnormality than for consistently identifying NSR for recordings with cardiac abnormalities. We don't pretend that our weights are the "correct" ones, but I'd argue that identifying a similar cardiac abnormality is better than missing a cardiac abnormality.

Your example shows that a "random" classifier may outperform a classifier than identifies NSR, but that doesn't mean that a random classifier has "acceptable" performance. It's difficult to compare scores because you generated random labels, but your score would not place well in the official results. We also used random classifiers to stress-test the evaluation metric during development and saw that random classifiers scored poorly in practice.

We're always open to idea on how to define an evaluation metric that better captures the problem that we're trying to solve, so please share feedback when we introduce this year's Challenge!

Best,
Matt
(On behalf of the Challenge team.)

https://PhysioNetChallenges.org/
https://PhysioNet.org/

Please post questions and comments in the forum. However, if your question reveals information about your entry, then please email challenge at physionet.org. We may post parts of our reply publicly if we feel that all Challengers should benefit from it. We will not answer emails about the Challenge to any other address. This email is maintained by a group. Please do not email us individually.