Data gathering and sampling method

[deniskok...@gmail.com]

Dear organizers,

Thank you for hosting this competition. 

I have some questions concerning data in SemEval-2023 Task-1 - Visual-WSD. Unfortunately, I was not able to find answers to these questions on competition page.
Could you please clarify the training data gathering process? What was the exact procedure of gathering and annotating data? This information might prove useful for creating better models.
How will the test data be sampled for evaluation period? Will it follow the same data gathering and annotating procedure? Will it share words, words' contexts or images with training sample?

Thank you in advance.

Best regards,

Denis Kokosinskii

[Iacer Calixto]

Hi Denis,

We will not share details about the data gathering or how we annotate the data. This is to make sure the competition is as fair as possible and that participants do not try to exploit this information to obtain artificially high results. Please note that we require users participating in our shared task to adhere to a fair data usage policy (see our website for details). All participants agree that they will not attempt to search the trial/training/test data using any search engine on the web, to reverse engineer the data generation process. Thanks and best of luck!

Iacer (on behalf of the organisers).

[Iacer Calixto]

Hi Denis,

I forgot to mention but we will share details about the data gathering process and annotation procedure after the competition has finished, we just won't do it while we are still accepting submissions. Hope that clarifies!

Best,

Iacer.