Question regarding evaluation time

[Tim Heuwinkel]

Good Afternoon,

 

we have a question concerning the valuation time of the MetaDL challenge. In the Rules it is stated that each submission receives 10 hours of compute time (2 hours per meta-dataset). We noticed that the timer that is displayed in the “My Submissions” Tab under “Total Compute Time” is accelerated by a factor of five, meaning 5 seconds of real time = 1 second of total compute time.

 

Why is that? Is the actual time limit 2 hours? Or does the evaluation of each meta-dataset maybe run in parallel?

 

Additionally, why does it display an execution time budget of 8:20:00 in the “My Submissions” Tab? I would have expected 10 hours. When Solutions are run that take longer than 8 hours and 20 Minutes, the remaining execution time displays a negative amount. Am I missing something here?

 

 

An answer to those question would help our team immensely.

 

 

Regards

 

Tim Heuwinkel

[Adrian EL BAZ]

Hello Tim! 

Thank you for your message. Indeed, each submission is run on the 5 meta-datasets, and each of these runs have a hard time limit of 2 hours. The time that is displayed next to your submission on the "My Submissions" tab is the current total execution time. Indeed, the execution is done in parallel if all workers are available. If some workers are not available during your submission (e.g. submissions from other participants are run on the workers), the total execution time displayed adapts accordingly.

Regarding your question about the time budget: there was a bug in the CodaLab platform and it is now fixed. You should be able to see and access the correct amount of execution time available (i.e. 10 hours instead of 08:20:00). Thank you for pointing out this issue. 

Best regards,

The organizing team

--
You received this message because you are subscribed to the Google Groups "metalearningchallenge" group.
To unsubscribe from this group and stop receiving emails from it, send an email to metalearningchal...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/metalearningchallenge/002001d79e6f%240e1032a0%242a3097e0%24%40heuwinkel.de.