Classes in the Test Set

[David Assaraf]

Dear Organizers of the competition, 

Thank you for the new scoring metrics and database. 

I wanted to clarify one point: since the weight matrix allowing us to compute the final score does not contain any 'other' label, can we assume that the test set will not contain any examples with a true label in the unscored pathologies? Or should we learn to our classifier to predict some 'other' label for unscored pathologies? 

Thank you, 

Sincerey, 

David 

[PhysioNet Challenge]

Dear David,

Like the training data, the test data may contain examples with labels that do not contain any scored classes. The Challenge metric will be unaffected by any classifier outputs on these examples.

Best,

Matt

(On behalf of the Challenge team.)

https://physionetchallenges.github.io/
https://physionet.org/

Please post questions and comments in the forum. However, if your question reveals information about your entry, then please email challenge at physionet.org. We may post parts of our reply publicly if we feel that all Challengers should benefit from it. We will not answer emails about the Challenge to any other address. This email is maintained by a group. Please do not email us individually. 

--
You received this message because you are subscribed to the Google Groups "physionet-challenges" group.
To unsubscribe from this group and stop receiving emails from it, send an email to physionet-challe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/physionet-challenges/f9ddb6ad-6e1d-4cdc-84ba-024e8c80c0f5o%40googlegroups.com.

[Yingjing Feng]

Dear Matt,

Thank you for your reply. 

As you said, the test data may contain examples with labels that do not contain any scored classes, if these examples have ground-truth diagnoses only in the unscored classes, it means that they are not sinus rhythm, but the current evaluation code will set the sinus rhythm to positive if all the predicted classes are negative. Will such cases affect the evaluation metric?

Many thanks,

Yingjing

On Saturday, July 4, 2020 at 4:27:06 AM UTC+2, PhysioNet Challenge wrote:

Dear David,

Like the training data, the test data may contain examples with labels that do not contain any scored classes. The Challenge metric will be unaffected by any classifier outputs on these examples.

Best,

Matt

(On behalf of the Challenge team.)

https://physionetchallenges.github.io/
https://physionet.org/

Please post questions and comments in the forum. However, if your question reveals information about your entry, then please email challenge at physionet.org. We may post parts of our reply publicly if we feel that all Challengers should benefit from it. We will not answer emails about the Challenge to any other address. This email is maintained by a group. Please do not email us individually. 

On Jul 3, 2020, at 9:03 PM, David Assaraf <david.as...@gmail.com> wrote:

Dear Organizers of the competition, 

Thank you for the new scoring metrics and database. 

I wanted to clarify one point: since the weight matrix allowing us to compute the final score does not contain any 'other' label, can we assume that the test set will not contain any examples with a true label in the unscored pathologies? Or should we learn to our classifier to predict some 'other' label for unscored pathologies? 

Thank you, 

Sincerey, 

David 

--
You received this message because you are subscribed to the Google Groups "physionet-challenges" group.

To unsubscribe from this group and stop receiving emails from it, send an email to physionet-challenges+unsub...@googlegroups.com.

[PhysioNet Challenge]

Dear Yingjing,

The Challenge scoring metric only scores recordings with one or more positive labels that belong to the scored classes. The metric only awards credit for positive classifier outputs that belong to the scored classes. If a recording only has positive labels that belong to the unscored classs, then we will not score this recording.

If a recording has ground-truth diagnoses only in the unscored classes, then the metric will not change the sinus rhythm class to be positive, and if the classifier returns positive diagnoses only in the unscored classes, then the metric will not change the sinus rhythm class to be positive. There is some confusion about what the evaluation code does when there are no positive classes, so I simplified this part of the code — if your classifier returns no positive classes (either scored or unscored), including the sinus rhythm class, then none of the classifier outputs are positive.

Best,

Matt

(On behalf of the Challenge team.)

https://physionetchallenges.github.io/
https://physionet.org/

Please post questions and comments in the forum. However, if your question reveals information about your entry, then please email challenge at physionet.org. We may post parts of our reply publicly if we feel that all Challengers should benefit from it. We will not answer emails about the Challenge to any other address. This email is maintained by a group. Please do not email us individually.

To unsubscribe from this group and stop receiving emails from it, send an email to physionet-challe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/physionet-challenges/5583f458-14a8-466c-accf-c11d07903b5ao%40googlegroups.com.