validation data und kfold

[Martin Sondermann]

Questions Regarding Model Submission and K-Fold Cross-Validation

Hello Challenge Organizers,

We have a few clarifying questions about the model submission process, particularly regarding cross-validation and model evaluation:

K-Fold Cross-Validation Approach

In our current implementation, we're using k-fold cross-validation (k=5) to train multiple models and gain more robust insights into our model's performance. However, this raises some questions about the submission requirements:

1. Multiple Models: Since k-fold cross-validation generates k different models, how should we handle the model submission? Are we expected to:

   • Choose a single "best" model from the k-fold splits?
   • Submit an ensemble or averaged model?
   • Submit all k models?

2. Output Folder Structure: The challenge instructions specify submitting models to an output folder. How would this work with multiple models from k-fold validation?

Validation Phase and Hidden Validation Set

We've noticed some nuances in the validation process that we'd like to clarify:

1. Internal Validation Split: In our training pipeline, we create an internal validation split for each training fold (either 80/20 or through k-fold cross-validation).

2. Challenge's Hidden Validation Set: Is the Challenge's hidden validation set completely separate from these internal validation splits?

3. Scoring Process: Could you confirm the scoring process:

   • Do you evaluate the model purely on the hidden validation set, disregarding our internal validation metrics?
   • Will the same approach be used for both the unofficial and official phases?

Model Training and Evaluation Workflow

To ensure we're following the challenge guidelines, we want to confirm our understanding of the workflow:

1. Train the model using our internal cross-validation strategy
2. Select/prepare a final model for submission
3. Your evaluation system will run inference on the hidden validation set to compute scores

We appreciate your guidance in clarifying these points to ensure we're developing our solution in alignment with the challenge requirements.

Thank you

[PhysioNet Challenge]

Dear Martin,

Thanks for the thoughtful questions. The quick answer is that cross-validation on the training set is a great way to understand and assess the generalizability of your method, and you and your team can decide how best to use the training set to develop your method.

For clarity, the training data is public, and the validation and test sets are private. The training, validation, and test sets do not overlap. In particular, your "internal validation set" is a subset of the training set and different from the actual private validation set. You can and should report cross-validated scores on the training set when you describe your results, but they may or may not approximate the scores on the validation and test sets, depending on how you perform cross-validation and the generalizability of your method to unseen data.

We can evaluate multiple entries from teams on the validation set during the unofficial and official phases of the Challenge, and we will evaluate at most one entry from each team on the test set after the official phase of the Challenge. You can decide what that entry will be -- whether it is a single model trained on the entire training set or a subset of the training set, an ensemble of models trained on the entire training set or subsets of the training set, or something else entirely.

Also, please remember that the training set contains data from different sources with strong and weak labels, so the thoughtful use of the training set is especially important this year.

Best,
Matt
(On behalf of the Challenge team.)

Please post questions and comments in the forum. However, if your question reveals information about your entry, then please email info at physionetchallenge.org. We may post parts of our reply publicly if we feel that all Challengers should benefit from it. We will not answer emails about the Challenge to any other address. This email is maintained by a group. Please do not email us individually.