Highest possible challenge score

[Jad]

Greetings,

I have a question about the highest possible score with the current challenge metric:

Let's say the evaluation dataset has more positive chagas cases that the 5% threshold. To my understanding, the highest possible challenge score would then not be 1, but rather be upperbounded by the ratio of the number of positives inside the threshold to the total number of positives in the dataset. In that case, it would also be possible that a perfect algorithm, that gets every prediction correct, receives the same score as an algorithm that misclassified a number of positive samples, as long as they aren't in the 5% threshold (To my understanding).

Is this behavior of the challenge metric desired? If yes, could it be possible, that the current validation dataset for the official phase has more positive chagas cases that the 5% threshold? 

Thank you for your time and help!

Best regards,

Jad Haidamous

[PhysioNet Challenge]

Dear Jad,

The 5% threshold for the Challenge evaluation metric reflects the serological testing capacity for Chagas disease. As you noticed, if the number of positive cases were to exceed the testing capacity, then even a "perfect" algorithm would not be able to identify all positive cases for serological testing. The prevalence rates in the training, validation, and test sets are approximately equal to the prevalence rate of Chagas disease in Brazil, which is the source of much of the Challenge data. Please see these discussions for more information:
https://groups.google.com/g/physionet-challenges/c/vrGuxgkr0UA/m/TMMnwp2iCwAJ
https://groups.google.com/g/physionet-challenges/c/EoDO6wbszxE/m/lfXqTF-dAAAJ

​​Best,
Matt
(On behalf of the Challenge team.)

Please post questions and comments in the forum. However, if your question reveals information about your entry, then please email info at physionetchallenge.org. We may post parts of our reply publicly if we feel that all Challengers should benefit from it. We will not answer emails about the Challenge to any other address. This email is maintained by a group. Please do not email us individually.