Question on submission evaluation time

[wenh06]

We received an email notification of an unsuccessful submission. But we found that the error was caused by code changes (newly added code, our master branch updates a few minutes just before submission) in another submission that was submitted about 20 hours after the submission mentioned in the notification email. So our question is, when will our  submission be evaluated after filling out the submission form? If a submission is evaluated many hours after it, maybe you are not evaluating the code that we want to be evaluated. Perhaps you can add an optional field in the submission form: the commit hash (short or long) or branch name, so that you can checkout to the correct version of code that is intended to be evaluated. Thank you!

[PhysioNet Challenge]

Yes, and you aren’t the only team to experience this either.

We pull from the repository at the time of processing, not at the time of submission. Now that the volume of submissions has increased significantly, there’s a significant backlog of submissions, so unfortunately some submissions are getting delayed.

Giving an option for the commit hash or the branch name is a good idea. Changing the way submissions get pulled would be out of scope for this year, but we will certainly consider implementing it for next year. Currently, the workaround some teams are using is to put the code for different submissions in different repositories.

Best,

James