Submission Format

[Wallace]

Hi John,

I have the following question about submission.

1. Where to check the format of our submissions before submit our runs?

2. Submit CCR runs and SSF runs in a single file, or two files?  eg. CCR.submit and SSF.submit,  or CCR_SSF.submit?

3. In CCR Task, an item is judged as level=0, -1. So CCR submission should exclude this item.

   But in SSF Task, we really find the a specific slot of entity, So SSF submission should include this item.

   Does this confliction?

Thanks,

Wallace

[John R. Frank]

> 1. Where to check the format of our submissions before submit our runs?

See Ellen's email to this list from August 11 with pointers to the
validation script inside the active participants area of trec.nist.gov

> 2. Submit CCR runs and SSF runs in a single file, or two files?  eg.
> CCR.submit and SSF.submit,  or CCR_SSF.submit?

Separate files.

> 3. In CCR Task, an item is judged as level=0, -1. So CCR submission
> should exclude this item.    But in SSF Task, we really find the a
> specific slot of entity, So SSF submission should include this item.  
>  Does this confliction?

You can include level=0 or -1 in a CCR submission-- the scorer will treat
them appropriately. Just make sure the relevance rating level is
indicated in the designated column of the tab-separated values.

When we started doing SSF, we thought that all *changing* slot values
would be in vital (rating=2) documents. Then we expanded SSF to include
more general slot fills, which changes this. In principle, SSF slot fills
can come from documents with *any* rating level, although a rating of -1
or 0 would imply that the system is either not sure the document mentions
the target entity at all or is certain that the document does *not*
mention the entity.

Are you saying that your slot filling system is able to learn things about
the entity from documents that do not mention the entity? That would be
very cool. It's logically allowable.

jrf

[Wallace]

Thanks, I got it.

在 2014年8月26日星期二UTC+8下午6时50分18秒，Wallace写道：