Updated Dataset and Deadline Extended

[LM-KBC]

Dear participants,

We have updated the released dataset (train/dev) with a more refined and complete list of object entities. Please do a `git pull` again to get a better evaluation.

Also, in lieu of the recent changes, the deadline for the final system and test dataset predictions submission has been extended to July 21. The hidden test dataset will still be released on July 11.

Good luck, and we look forward to your submissions!

Best,
Sneha

[Thiviyan Thanapalasingam]

Dear Sneha, 

I understand that you are planning to release the hidden test dataset on the 11th of July. As far as I know, most ML competitions do not release the test datasets. This is because the final performances are usually measured against the test dataset and therefore, keeping the test dataset hidden would prevent candidates from using it (either accidentally or intentionally) when tuning hyperparameters. 

Are there any reasons why the organisers of LM-KBC decided to release the test dataset? Wouldn't it give unfair advantage to some candidates?

Best, 

Thiviyan

[LM-KBC]

Dear Thiviyan,

Thanks for your message. 

On July 11, we will only release the test subject entities and not the ground truth object entities. Since the challenge comes with an open track (track 2) in which any LM can be used, we plan to release the subject entities for computational feasibility. 

Best,

Sneha

[Thiviyan Thanapalasingam]

Dear Sneha, 

Thank you for your prompt response!  Does that mean that the test set contains entities that are not present in the training and validation set?

Best, 

Thiviyan

[LM-KBC]

Yes, that's correct. The relations remain the same though.

Sneha