results task 1 vs task 2

[Jeroen Vuurens]

Hi,

We only submitted results for task 1 and also received results for task 2 which are slightly lower in absolute numbers. For scoring the metrics for task-2, did you simply use the results for task 1 and then for scoring task 2 only consider the sentences that were part of the pre-filtered stream? 

Thanks, Jeroen

[Jeroen Vuurens]

And additionally, how was expected gain computed for the team.all.tsv file, because without calling numbers the expected gain in the .task1 and .task2 files is roughly the same but the expected gain in the .all file is only roughly half of that (as if the results were simply merged without removing duplicates). Is that correct?

[Matthew EA]

Apologies, it looks like I did accidentally put your files into task 1 and task2. And yes, the task 2 numbers would be lower because it would exclude matches outside the task2 dataset. Please ignore the results in your task2 file. Unfortunately that also means the task2 averages are slightly off.

Cheers,

Matthew

[Jeroen Vuurens]

Hi Matthew,

Thanks for answering my question. I was just curious about the difference between the two, if using the entire set in timely order is 'allowed information' for task 2, then using the results for task 1 by excluding the matches outside the filtered set seems valid to me. However, regarding the second part of my question, will you then just use the .task1 and .task2 files for the TREC report, or also use what is reported in the '.all' file?

Thanks, Jeroen

--
You received this message because you are subscribed to the Google Groups "temporalsummarization" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trec-ts+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Matthew Ekstrand-Abueg]

We will most likely just be using the individual track files.