FAKBA1: information from the future?

[Stuart Mackie]

Hi,

Is the "Google Freebase Annotations of TREC KBA 2014 Stream Corpus, v1 (FAKBA1)" considered to be information from the future, for the purposes of TREC TS?

This dataset provides entity linking annotations (to Freebase) over documents within KBA 2014, more information:

http://trec-kba.org/data/fakba1/
http://www.searchenginecaffe.com/2015/02/google-research-entity-annotations-of.html

I know there are already NER annotations (from BBN's Serif) present in the KBA 2014 corpus metadata.

In short, FAKBA1 is an interesting dataset, it might be useful for TREC TS, but is it information from the future?

thanks,
Stuart.

[Fernando Diaz]

Stuart,

This is a fair question and I'm looking into how the annotations were built.  The risk is that some annotations/entities may not have been recognized at simulation time.  This leaks information from the future and hurts generalizability.

F

---

From: tre...@googlegroups.com <tre...@googlegroups.com> on behalf of Stuart Mackie <s.mac...@research.gla.ac.uk>
Sent: Wednesday, July 8, 2015 4:40 AM
To: tre...@googlegroups.com
Subject: [TREC-TS] FAKBA1: information from the future?

 

--
You received this message because you are subscribed to the Google Groups "temporalsummarization" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trec-ts+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Fernando Diaz]

Stuart,

After some discussion, the organizers have decided that participants can use FAKBA1 with the following conditions,

1. ​this is will an "external evidence run"
   
2. ALL rows in the FAKBA1 dataset with Freebase identifiers that did NOT exist before event start MUST be removed/not considered.

Please let us know if you have any questions.

Fernando Diaz

---

From: tre...@googlegroups.com <tre...@googlegroups.com> on behalf of Fernando Diaz <fd...@microsoft.com>
Sent: Wednesday, July 8, 2015 10:19 AM
To: Stuart Mackie; tre...@googlegroups.com
Subject: Re: [TREC-TS] FAKBA1: information from the future?