Clarification on Threshold Optimization in compute_challenge_score

[aditya nagori]

Hello Challenge Team,

I’m implementing the “sensitivity at 5% positive‐rate” metric for our submission using compute_challenge_score function. I’d like to confirm exactly how your compute_challenge_score(labels, outputs, max_fraction_positive=0.05) function is used during scoring: 

• Does it internally optimize the decision threshold on the held-out (test) set to achieve the 5% positive-rate guideline?

• Or, does it simply compute sensitivity at a fixed threshold that participants must determine beforehand (e.g., on a validation set, maybe during training) and supply to the scoring pipeline?

Thank you for clarifying whether threshold selection happens inside the challenge scorer or must be frozen in our code submission. 

Best, 

Aditya

[PhysioNet Challenge]

Dear Aditya,

Yes, the evaluation code (the compute_challenge_score function) determines the smallest threshold so that at most 5% of the encounters with the highest scores (the Chagas probability score) from your model are positive and the remaining encounters are negative. You do not need to provide this threshold.

This post has related information about the evaluation metric and the 5% threshold:
https://groups.google.com/g/physionet-challenges/c/vrGuxgkr0UA

We are looking forward launching the official phase of the Challenge in the coming days; please stay tuned!

Best,
Matt
(On behalf of the Challenge team.)

Please post questions and comments in the forum. However, if your question reveals information about your entry, then please email info at physionetchallenge.org. We may post parts of our reply publicly if we feel that all Challengers should benefit from it. We will not answer emails about the Challenge to any other address. This email is maintained by a group. Please do not email us individually.