Evals on AVA Active Speaker "secret" test set

[Sourish Chaudhuri]

Hello AVA folks,

                         In the run-up to CVPR submission deadlines, we've received some reasonable requests from users of the dataset, looking to add performance metrics on the secret test set used in the AVA Active Speaker track of the ActivityNet workshop.

                         Since the ActivityNet evaluation server is closed now, we're willing to help run that eval for users who would like this number. However, to avoid users starting to overfit models on this test set, we're restricting the absolute number of evals a user/team may run to 2, at most.

                        While the recent requests have been motivated by in-progress submissions to CVPR, we recognize that different people will have different venues or measures of progress in mind. As such, I'm willing to take requests for evals between now and December 31, 2020. 

                        If you'd like an evaluation for your model generated predictions on the secret test set for the AVA ActiveSpeaker task, please follow the following guidelines:

1. Please send me an email (so...@google.com) with the file containing predictions in the same format as described on the ActivityNet page. Please use "AVA Active Speaker test set eval request" as the email subject. 

2. Please note your (possibly, tentative) paper title and list of authors in your email, to help track evals run for you. [We recognize these may not be final already, please list folks as per your best guess today.]

3. Please do not submit more than 2 requests total per research team.

4. Please expect a 2 working day turnaround time for all requests. 

I will respond with the mAP number on the eval set, as used in the ActivityNet challenge leaderboard. 

                          Please reach out to me if you have any thoughts on this proposed process.

Thanks,

 Sourish