Evaluation question

[Vibou]

Hi John,

I just wanted to know how are going to be evaluated documents that we found and that are not part of the test collection ? 

• As FALSE POSITIVE (not in testCollection) ? 
  
• Only those we found as vital and that are not in test collection are considered as FALSE POSITIVE (not in test collection && classifiedAsVital) ? 
• Otherwise it would be TRUE NEGATIVE (if not in test collection && classifiedAs non-vital)? 
  

Please, tick the write answer if there is one :D 

I m trying to evaluate my system but I want to do it accordingly to what is going to be done for KBA14 :) 

[John R. Frank]

Hi Vibou,

By "test collection" do you mean the ...kba-filtered corpus of 20,494,260
StreamItems?

Any stream_id that is included in a run submission that is not in the
...kba-filtered corpus will be ignored by the scorer and not treated as
any of FP, TP, TN, TP

We may use them in expanding a future corpus after 2014.

Does that answer your question.

jrf

On Fri, 22 Aug 2014, Vibou wrote:

> Hi John,
> I just wanted to know how are going to be evaluated documents that we found and that are not part of the test collection ? 

> * As FALSE POSITIVE (not in testCollection) ? 
> * Only those we found as vital and that are not in test collection are considered as FALSE POSITIVE (not in test collection && classifiedAsVital) ? 
> * Otherwise it would be TRUE NEGATIVE (if not in test collection && classifiedAs non-vital)? 

> Please, tick the write answer if there is one :D 
>
> I m trying to evaluate my system but I want to do it accordingly to what is going to be done for KBA14 :) 
>

> --
> You received this message because you are subscribed to the Google Groups "TREC-KBA" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to trec-kba+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>