Difference in Deadlines

[Praveen Kumar]

Hi John,

   Can you tell me what is the difference between regular and extended deadline?

Thanks,

Praveen

[Jingang Wang]

Hi Praveen,

I held the same concerns before.

The KBA 2013 official site says：

As an incentive to submit your CCR runs by the Regular Deadline we have coordinated with TAC KBP for an exciting joint experiment called "KBX"

The KBP organizers will endeavor to use top scoring KBA runs submitted by the Regular Deadline as input corpora to KBP Cold Start.

If feasible, KBX will evaluate end-to-end performance of combining KBA+KBX algorithms that filter a stream of ~10⁹ documents and emit a knowledge base about a specific group of related entities.

Note: KBX is an experiment. It might get snarled up in a few ways: the KBA runs might not have enough "small town" entities and TAC-style relations to meet Cold Start's needs, or it might have so many documents that it exceeds the KBP assessor budget. Also, we are still structuring its scientific questions and metrics.

That said, neat things often appear at interfaces, so we feel compelled to try. If you feel similarly compelled, then submit KBA CCR runs by the Regular Deadline in June. 

So I think if you are not interested in the KBX, you could ignore the regular deadline.

--
You received this message because you are subscribed to the Google Groups "TREC-KBA" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trec-kba+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Wang Jingang(王金刚)

Ph.D. Candidate at  

Lab of High Volume Language Information Processing & Cloud Computing

School of Computer Science

Beijing Institute of Technology

Beijing 100081

P.R China

[John R. Frank]

> Can you tell me what is the difference between regular and extended
> deadline?

Good question.

The Regular Deadline is in about a month, and is determined by the SSF
annotation schedule at NIST. This year, we have a second round of NIST
assessor time from mid-June to mid-July.

We are using *pooled* runs for SSF annotation. Unlike CCR, for which we
generate reasonably exhaustive PRE-hoc ratings, the recall of SSF
annotation depends more on what you submit. SSF is judged POST-hoc.

While one can submit SSF runs for the Extended Deadline, these SSF runs
will not be included in SSF judging period conducted in June-July. SSF
runs submitted for the Extended Deadline will be scored against the
judgments generated from showing the assessors the pooled SSF runs
submitted by the Regular Deadline.

Time permitting, we will also use the June-July round of NIST assessor
time to generate more CCR annotations by pooling CCR runs that are
submitted by the Regular Deadline.

This means that everyone has a big incentive to get something (both CCR
and SSF) submitted by the Regular Deadline, and then continue improving
until the Extended Deadline.

Please let us know if you have any questions about this. Happy to explain
more.

jrf

[Anshul Mittal]

Hi John,

If we are unable to submit anything by the regular deadline, can we still submit a run by the extended deadline?

Anshul

[John R. Frank]

> Hi John,If we are unable to submit anything by the regular deadline, can

> we still submit a run by the extended deadline?

Yes. You can submit runs before either or both.

We will not release the full annotation data until after the "Extended"
(aka final) deadline on August 28th.


There are two reasons to try to get some runs in by this Wednesday:

1) helps ensure recall for SSF annotation.

2) we wil attempt to construct a "KBX" corpus for the TAC KBP evaluation
using this first round of runs. KBX is described here:

https://groups.google.com/d/msg/trec-kba/ayImlJTqQhs/EfjpRKkFpoAJ


John