Question concerning reporting of Test Score in preprint

[Allan Moser]

The preprint template and CinC paper checklist say to include the results on the test set. However, my understanding is that we will not know our score on the test set prior to the deadline for submitting the preprint.  Should we just include the final results on the validation set as shown on the leaderboard?

Thank you,
Allan Moser

[PhysioNet Challenge]

Dear Allan,

Yes, please include your scores on the validation set in the preprint. We will share the scores on the test set shortly after the conference, so you can't include them in your preprint, but please do include your scores on the test set in the final paper.

Many teams have multiple successful entries on the validation set. You can report the scores on all of these entries, but we will only evaluate one of them on the test set (whichever entries you chose, or the highest scoring entry on the validation set for teams that didn't choose a specific entry), so please focus on that entry in your manuscript.

Please report your Challenge score, which we will use to rank the entries, but you can include additional scores in your manuscript to provide additional context to your method; the template has more information on how to report scores in a consistent way.

Best,

Matt
(On behalf of the Challenge team.)

Please post questions and comments in the forum. However, if your question reveals information about your entry, then please email info at physionetchallenge.org. We may post parts of our reply publicly if we feel that all Challengers should benefit from it. We will not answer emails about the Challenge to any other address. This email is maintained by a group. Please do not email us individually.