Missmatch between unique <s,p>

Dimitris Alivas

unread,

Jul 19, 2022, 9:14:57 AM7/19/22

to LM-KBC

Dear organizers,

We noticed that the relation counts differ between website and the actual file (dev.jsonl). According to the website it should include 50 unique <subject, entity> pairs per relation, however it looks like for CountryBordersWithCountry and CountryOfficialLanguge we get 47.

Could you kindly confirm and let us know whether this is an issue moving forward?

Kind regards,

Dimitrios

LM-KBC

unread,

Jul 19, 2022, 9:27:41 AM7/19/22

to LM-KBC

Hi Dimitrios,

There are less than 200 countries in the world, hence we decided to keep 100 in the train, 50 in the test, and the rest in dev. This ensures consistency in the number of entities present in the test set for a fair evaluation. Since the dev set is typically used for parameter tuning, this shouldn't be an issue.

Best,
Sneha

Dimitris Alivas

unread,

Jul 19, 2022, 9:34:49 AM7/19/22

to LM-KBC

That makes sense!

Thank you for the prompt response.

Best,
Dimitrios

Reply all

Reply to author

Forward