number of submissions

[Fernando Diaz]

We will be following same policy as last year,

There is currently no limit on the number of runs nor the number of updates per run which a group can submit.

The runs should use a positive numerical index (lower is more important) as part of the run id if you are planning on submitting lots of runs, such that we can limit if we find we have too many submissions. I
imagine that at least the first 4 runs will be evaluated, but likely more than that if you choose to do so (especially if they have high update overlap).  

For the updates, they will be pooled and cut off based on confidence value for each run if there are too many (higher is better). Confidence values need not be comparable between runs or teams, but should be comparable within a run.  We anticipate approximately the top 60 updates being evaluated from each run-event pair.  

Please check that you are using the correct sentence indexing and format.  We will not attempt to fix incorrectly formatted submissions.

Let us know if you have any questions.

[luk...@udel.edu]

Thanks for the information. That's very helpful.

在 2015年8月27日星期四 UTC-4下午1:22:35，Fernando Diaz写道：

[Shruti Bhargava]

Hi,

Is it alright if the run covers the updates for only some topics? 

Also, how is the run id decided? and is the team id same as the group id?

Thank You.

Regards,

Shruti Bhargava

[Matthew Ekstrand-Abueg]

I believe you are required to have at least one update per topic (by the submission system). If you do not wish to provide updates for a topic, you can simply use a dummy update. But final scores will be averaged across all topics.

You determine your run ids. They should end with a positive integer, with lower numbers indicating higher priority for evaluation (i.e. we may only evaluate the 4 runs with lowest ending numbers, depending on number of submissions). If your runs are not numbered, which ones are chosen for evaluation may be random.

Yes, team and group ids are the same.

--
You received this message because you are subscribed to the Google Groups "temporalsummarization" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trec-ts+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Fernando Diaz]

You can submit a run with missing events but you will get pretty low scores.  

The runid is your decision and can be something like "<group id><priority>" where <priority> is an integer as described earlier.  

---

From: tre...@googlegroups.com <tre...@googlegroups.com> on behalf of Shruti Bhargava <shrutibh...@gmail.com>
Sent: Tuesday, September 1, 2015 11:53 AM
To: temporalsummarization
Subject: [TREC-TS] Re: number of submissions

 

[Shruti Bhargava]

Hi,

For topic 31, the documents given for the filtered version of the corpus(TREC-TS-2015-F) are outside the range of timestamps given for the topic.Our approach is giving very few updates for the restricted urls within time range. While it might be late to generate updates for new set of urls, could the validation script account for the urls outside timestamp ?

Thanks for the replies,

Shruti Bhargava

[Fernando Diaz]

1. we will only judge updates in the evaluation time period. 

2. you should not be looking at your system output on the evaluation events to tune for, eg, number of updates. 

[Richard McCreadie]

On the topic 31 question:

You are correct, in that the filtered document stream for topic 31 lasts for some time after the end date of the event in the topic file (we had originally envisaged that the topic would have a longer timespan, but shortened it during topic development).

Under the rules for the task, only documents from within the timerange provided in the topic file should be considered. So, you should stop your system when the end timestamp is reached (or filter out any sentences returned after that timestamp).

Best Regards,

RichardM