Rules submission

[Arnaud Champetier]

Dear Physionet Team,

Once again, I’m thankful for the organisation you provide.

My question is regarding the submission. As an example, let's suppose that I plan to train a model using a 10-fold cross-validation method and take the average predictions as the final predictions.
Can I submit the code to train (fine-tune) the models for 1 epoch, using a pretrained model on 9 epochs on the dataset from the competition? Is that allowed, or do you require that the 10 epochs must fit the training time?

Kind regards,

Arnaud

[PhysioNet Challenge]

Dear Arnaud,

We require each team to submit working training code that trains a model on the provided training data within the provided resource constraints.

In your example, you asked (as I understood it) if you could train your model for 9 epochs on your machine, include the trained model in your submission, and continue training the model for 1 epoch on our submission system. This approach appears to attempt to circumvent the resource constraints, and it does not appear to learn much from the provided training set. We perform various checks to ensure that the final entries include working training code, and this approach would fail our checks.

We have not shared the details of our checks to avoid incentivizing the minimum to pass them, but the general principle of whether your model actually learns from the training data applies, and should be relatively easy to check. If the training data changes, then the model should also change. Small changes to the data may result in small changes to the model, larger changes to the data may result in larger changes to the model, and so on. In the future, if a new dataset were to become available, or new or improved labels for an existing dataset, then working training code allows your model to improve with the available data.

Best,
Matt
(On behalf of the Challenge team.)

Please post questions and comments in the forum. However, if your question reveals information about your entry, then please email info at physionetchallenge.org. We may post parts of our reply publicly if we feel that all Challengers should benefit from it. We will not answer emails about the Challenge to any other address. This email is maintained by a group. Please do not email us individually.