CEA, CTA, CPA Evaluation

[Brice FOKO]

Helle dear Organizers,  

Please, can you give us the process to follow in order to evaluate ourselves Precision and Recall for our CEA, CTA, CPA results ? This should help us to improve our different approaches for this challenge. 

Thank you in advance.

Kind regards,

[Vincenzo Cutrona]

Hi!

The code we use for evaluating your submissions is the same as you can find along with each dataset (e.g., CEA_DBP_Evaluator.py and CTA_DBP_Evaluator.py for ToughTablesR2-DBP). You can validate your solution on the given "Valid" table set. Every week, we run the same code to evaluate your submission against our "Test" ground truth.

Best,
Vincenzo

[Brice FOKO]

Good evening dear organisers,
Thank you for your reply.

Please, in the valid cta and cea, we have some lines with multiple annotations.
For example, in cea_gt.csv of ToughTables WD, we have this: "8QA9EYPI", "135", "0", "http://www.wikidata.org/entity/Q355907 http://www.wikidata.org/entity/Q12260929 http://www.wikidata.org/entity/Q95376520"

I'm a little confused because the challenge statement specifies a single annotation for each target, so I'm wondering:
- Is this correct?
- Shouldn't each of the lines have a single URI instead of several as above?

Also, I hope that at the end of this challenge, it will be possible to get your "Test" ground truth.

Kind regards

[Vincenzo Cutrona]

Hi!

Then, while submissions must contain at most 1 entity per each target, the GT may contain multiple ones. An annotation in the submission file is valid if it is one of the entities listed in the GT file.
In your specific example, your annotation is right if you annotate cell 135,0 in table 8QA9EYPI with Q355907 OR Q12260929 OR Q95376520. All these 3 entities are valid because the URIs point to the same entity (József Antall, Prime Minister of Hungary, 1990-1993).

Anyway, the evaluation code already handles this process for you.

We usually release Test datasets at the end of the challenge.

Best,
Vincenzo

[Brice FOKO]

Hi organizer,

Thank you for your reply.

--
You received this message because you are subscribed to a topic in the Google Groups "Sem-Tab Challenge" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sem-tab-challenge/wPaVprBXFOg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sem-tab-challe...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/sem-tab-challenge/01694425-51e0-4c57-8e1c-c68928eafce5n%40googlegroups.com.