compute_challenge_score function

[Camilo Valderrama]

Hi,

Our team is trying to understand how the challenge score is computed. On line 67 of the file "evaluate_model.py", the function compute_challenge_score is called as follows:

 challenge_score = compute_challenge_score(labels, probability_outputs) 

The signature of function compute_challenge_score is defined in the helper_code file as: def compute_challenge_score(labels, outputs, max_fraction_positive = 0.05),

where it is clear that the default max_fraction_positive is a 0.05 by default. 

We are wondering if, given that max_fraction_positive is a relevant argument for computing the score, we could modify the call of the compute_challenge_score function in the evaluate_model.py file as: 

challenge_score = compute_challenge_score(labels, probability_outputs, proportion_chagas_cases) 

This will allow the compute_challenge_score to know the proportion of positive cases so that the score can be calculated properly. The latter is because using the default value (max_fraction_positive=0.05) could produce a wrong estimation if there are more than 5% cases in the test labels.

We are confused if the modification of the call of compute_challenge_score  is allowed.

I would appreciate any feedback you could provide us.

Thank you.

Best,

Camilo

[PhysioNet Challenge]

Dear Camilo,

This parameter determines the number of encounters that can be referred for serological testing for Chagas disease. Unfortunately, it is not determined by the prevalence rate of Chagas disease in the data but by the serological testing capacity for Chagas disease.

You can think about your algorithm as screening or triaging patients for Chagas disease.
If the testing capacity exceeds the number of Chagas-positive patients, then every Chagas-positive patient could be tested if an algorithm were to prioritize them well enough. Conversely, if the number of Chagas-positive cases exceeds the testing algorithm, then some Chagas-positive patients could not be tested no matter how well the algorithm were to prioritize them.

This issue is a real-world constraint, and the choice of this parameter is an estimate of the testing capacity from our clinical collaborators in Brazil. (The real number depends on many factors, and varies geographically, etc., but 5% is a reasonable estimate.) We have included the testing capacity as a parameter to the scoring function because it is a parameter, e.g., the local serological testing capacity could change over time or in different settings, but we want it to be realistic and consistent between teams, so different teams cannot choose different values of the parameter.

We are looking forward to making many changes for the official phase of the Challenge, including to the scoring function, so observations and feedback are especially helpful right now between the unofficial and official phases of the Challenge. Relatedly, some teams received high leaderboard scores during the unofficial phase, but the official phase will bring many changes, so no one should be discouraged ahead of the official phase -- please stay tuned.

Best,

Matt
(On behalf of the Challenge team.)

Please post questions and comments in the forum. However, if your question reveals information about your entry, then please email info at physionetchallenge.org. We may post parts of our reply publicly if we feel that all Challengers should benefit from it. We will not answer emails about the Challenge to any other address. This email is maintained by a group. Please do not email us individually.