Is there any baseline for our reference?

62 views
Skip to first unread message

Albert ZHU

unread,
Apr 22, 2021, 10:29:56 PMApr 22
to OpenNetLab Challenge
Dear organizers,

Thanks for this grand challenge.

OpenNetLab’s website is not available at this time. So, I wonder is there some updates about the Evaluation Criteria?

What's more, could you please tell me if there is some evaluation baseline for our reference?

Thank you!

Best regards,
Albert



Zhixiong

unread,
Apr 26, 2021, 9:04:47 PMApr 26
to OpenNetLab Challenge
Hi Albert,

We will use VMAF and DNSMOS to do video and audio metrics. We also consider some delays to measure if your model can deliver the frames in time. 

OpenNetLab Challenge

unread,
Apr 26, 2021, 11:25:23 PMApr 26
to OpenNetLab Challenge
Dear Albert,

    We have an challenge environment (https://github.com/OpenNetLab/Challenge-Environment) that includes the evaluation system. Actually, we will have two evaluation system, online and offline.
    The online system will just provide a crude score which will not be used as rank but used to tell the contestant the submittal is accepted to our system. This part have been published in the github repository.
    The offline system will be used as rank, it's almost same as the online evaluation system but will add more testcases and complex frame alignment algorithm. It will also be published in this repository about the end of May.

      I hope my answer can help you. If you have more concerns or question, please don't hesitated to contact us.

Thanks,
Ze

Reply all
Reply to author
Forward
0 new messages