Test Data and Leaderboard

[VisDA 2021 Participants]

Dear participants,

Thank you for your continued interest in the VisDA-21 competition, and we are happy to announce that the test phase is live!

Test images can be found here: http://csr.bu.edu/ftp/visda/2021/test_data.zip

And an example submission file with correct file order (alphabetical) can be found here: http://csr.bu.edu/ftp/visda/2021/test_example_submission.zip

The test leaderboard can be found at https://competitions.codalab.org/competitions/33396 under “Results > Testing Data Released”. Teams will be ranked according to their average ranking in target accuracy and detection AUROC, with accuracy as a tie-breaker (see http://ai.bu.edu/visda-2021/#rules for details). Each team is allowed to make at most one submission a day and at most five submissions total. If different team members use different codalab accounts, the same rules apply to their cumulative submission counts. In order to qualify as winners, a team must make their final result public on the test leaderboard by Oct 10th, 11:59 am EST. Shortly after, we will request winning teams to provide four+ page reports describing their method, and submit their code so that we could verify their results within the next two days. Reports must contain a checklist section (see below) verifying that the proposed solution fits into challenge rules. All winners’ reports will be posted on the challenge website and authors will be given an opportunity to provide updated “camera-ready” versions of their reports later. Open sourcing the code is optional but is highly encouraged. 

If your submission did not make it to top-3 but is particularly efficient in terms of resources required for training, please consider submitting an entry into our energy efficiency “honorable mention” track: https://forms.gle/ArxjHfcSrfYuQjXj9 

If you have any questions, please contact us via visda-or...@googlegroups.com .

Good luck and have fun!

VisDA-21 team

http://ai.bu.edu/visda-2021/

Report checklist:

1. Supervised Training: Teams may only submit test results of models trained on the source domain data. To ensure equal comparison, we do not allow training on any other external data or any form of manual data labeling.

2. Unsupervised training: Models can be adapted (trained) on the test data in an unsupervised way, i.e. without labels.

3. Model size: To encourage improvements in universal domain adaptation, rather than general optimization or underlying model architectures, models must be limited to a total size of 100 million parameters.

4. Ensembling: Ensembling is now allowed, but each additional forward pass through a parameter will count towards the total number of parameters allowed (100M). For example, an ensemble that passes through a 50M model twice will count as 100M towards the parameter cap.

5. Energy efficiency: Teams must report the total training time of the submitted model which should be reasonable (which we define as not exceeding 100 GPU days of V100 16GB version).

[VisDA 2021 Participants]

Dear participants,

As the final submission date (Oct 10th, 11:59 am EST) slowly approaches, we would like to remind you that:

1. Due to technical limitations, the codalab test leaderboard currently ranks submissions based only on the target ACC. Please disregard this ranking - we will use the average of ACC and AUROC ranks (the "< Rank >" column) as the primary final ranking score and resolve ties using ACC rank, as described in official rules: http://ai.bu.edu/visda-2021/#rules .

2. Participants shall not use any additional data or pre-trained feature extractors beyond those pre-trained on Imagenet-1K.

3. Participants shall not use the validation set to train the model used during test phase.

4. Participants shall not use exact corruptions used in ImageNet-C (listed here: https://github.com/VisionLearningGroup/visda21-dev#corruption-blacklist ).

5. In order to qualify as winners, a team must make their final result public on the test leaderboard by Oct 10th, 11:59 am EST. 

Good luck and have fun!

VisDA-21 team

visda-or...@googlegroups.com

http://ai.bu.edu/visda-2021/