Unofficial phase model test

[Gihun Joo]

Hello :)
When submitting in the unofficial phase, will the evaluation be based on the actual model training and testing, or will the score be based on the pre-existing saved model?

[PhysioNet Challenge]

Dear Gihun Joo,

Good question!

We want to evaluate the model produced by your training code, so it is important to write your code in a way that allows us to evaluate the model produced by your training code (without needing to edit your code to do so).

For each submitted entry, we will run the following commands:

MATLAB:
(1) train_model(training_data, model)
(2) run_model(model, test_data, test_outputs)
(3) evaluate_model(labels, outputs, scores.txt)

Python:
(1) python train_model.py training_data model
(2) python run_model.py model test_data test_outputs
(3) python evaluate_model.py labels outputs scores.txt

Please write your code so that these commands (1) use the training set to train a model, (2) use this model to make predictions on the validation or test set, and (3) use these predictions to evaluate your model. Please do not write your code so that these commands do not train a model, or so that they train a model but later load another model. If we find that a team's code does not learn from the training set or attempts to use a model that does not learn from the training set, then we will not be able to evaluate the code on the test set.

We impose this requirement on teams to ensure that their code more fully implements their approach. If you have questions about this requirement, or are concerned about the application of this requirement to your approach, then please ask.

You do not need to upload a pre-trained model with your code. We do allow transfer learning with pre-trained models, but your code must actually perform transfer learning if it uses a pre-trained model. Please see this FAQ post and related questions in the forum for details:
https://moody-challenge.physionet.org/faq/#external-data

Best,
Matt
(On behalf of the Challenge team.)

Please post questions and comments in the forum. However, if your question reveals information about your entry, then please email info at physionetchallenge.org. We may post parts of our reply publicly if we feel that all Challengers should benefit from it. We will not answer emails about the Challenge to any other address. This email is maintained by a group. Please do not email us individually.