TREC KBA important details --- April 8th is coming!

[John R. Frank]

KBAers:

KBA starts in about four weeks. On April 8th, we plan to release
target_ids of entities, training data for CCR, and target slot names for
SSF.

We have updated http://trec-kba.org/trec-kba-2013.shtml

This year, we are including entities from both Twitter and Wikipedia, and
we have revised the CCR relevance levels to be more temporally oriented:
http://trec-kba.org/trec-kba-2013.shtml#relevance


There are two deadline dates for KBA:

1) Regular Deadline: Wednesday, June 12th

2) Extended CCR Deadline: Wednesday, August 28th


There are two filtering tasks in KBA 2013:

1) cumulatative citation recommendation (CCR)

2) streaming slot filling (SSF)


Runs for both tasks are due by the Regular Deadline. We will continue to
accept CCR submissions through the Extended Deadline. We will not accept
SSF submissions after the Regular Deadline, because SSF will be judged
post-hoc with pooled assertions from run submissions. Like last year, CCR
will be scored using pre-hoc ratings judgments -- some of which will be
provided as training data when the target entities are announced.


** KBX incentive **

As an incentive to submit your CCR runs by the Regular Deadline we have
coordinated with TAC KBP for an exciting joint experiment called "KBX"

http://trec-kba.org/trec-kba-2013.shtml#kbx

KBX: the KBP organizers will endeavor to use top scoring KBA runs
submitted by the Regular Deadline as input corpora to KBP Cold Start.

If feasible, KBX will evaluate end-to-end performance of combining KBA+KBX
algorithms that filter a stream of ~10^9 documents and emit a knowledge
base about a specific group of related entities.

I must stress that KBX is an experiment. It might get snarled up in a few
ways: the KBA runs might not have enough "small town" entities and
TAC-style relations to meet Cold Start's needs, or it might have so many
documents that it exceeds the KBP assessor budget. Also, we are still
structuring its
scientific questions and metrics.

That said, neat things often appear at interfaces, so we feel compelled to
try. If you feel similarly compelled, then get your KBA CCR runs in June.



Regards,
KBA Organizers