dry run

[Pierre-G Aublin]

Dear organizers,

Compared to some previous years challenges, is there a possibility of dry runs ?
I mean submitting a code to get a feedback on whether it runs on the submission system, and how long it takes to train/run, but not to get the scored entry ?

Yours faithfully
Aublin Pierre.

[PhysioNet Challenge]

Dear Aublin Pierre,

Unfortunately, we do not currently support dry runs.

As you noted, the previous submission system (before 2019) allowed teams to submit entries as “dry runs” that would notify teams if their entry was successful or not but would not provide the score on the hidden data or count again their entry limit, which prevents overfitting.

The current submission system uses containerized entries, which makes it easier to know if your team's entry will be successful or not. Once your team has a successful entry, it is easy to check that your team has not introduced breaking changes for future submissions by running the entry locally with the public training data. In any case, unsuccessful entries do not count against your team's entry limit.

Dry runs also incur human and computational time, so there are also practical reasons why we do not currently support dry runs, but we may be able to support this sort of feature in the future.

Best,
Matt
(On behalf of the Challenge team.)

https://PhysioNetChallenges.org/
https://PhysioNet.org/

Please post questions and comments in the forum. However, if your question reveals information about your entry, then please email challenge at physionet.org. We may post parts of our reply publicly if we feel that all Challengers should benefit from it. We will not answer emails about the Challenge to any other address. This email is maintained by a group. Please do not email us individually.