TREC 2015 Temporal Summarization submission system now open
ellen.v...@gmail.com
The submission deadline for Temporal Summarization track runs is September 3. A submission deadline of Sept 3 effectively means that you must successfully submit your run prior to NIST staff arriving at NIST on Sept 4---basically by 6:00am EDT on Sept 4. There is no possibility of this deadline being extended. NIST assessors start their task early the next week, and Monday Sept 7 is a US holiday, so all preparations for the assessors must be complete on Friday, Sept 4.
To submit a run, fill in the submission form by answering questions that describe the run and specify the file to upload. [Because the submission forms reside on a different machine outside the NIST firewall from the rest of the TREC website, you will need to give the active participants' login and password to first access the link to the form and then again to access the form itself (the first time you access the form).] After you click submit, the submission system will run a validation script that will test the submission file for various kinds of formatting errors. A pointer to this script is on both the active participants' track page and in the 'Tools' section of the active participants' web site. Over the years NIST has found that strict checking of the "sanity" of an input file leads to far fewer problems down the line as it catches a lot of mistakes in the run at a time that the submitter can actually correct them. You are strongly encouraged to use the script to test your submission file prior to submitting the file to NIST. Invoke the script giving the run file name as the argument to the script and an error log file will be created. The error log will contain error messages if any errors exist, and will say that the run was successfully processed otherwise. If any errors are found by the script at the time the run is submitted, the submission system will reject the run. Rejected runs are not considered to be submitted; indeed, no information is retained about rejected runs. Submitting a run through the submission system is the only acceptable way to send a run to NIST. In particular, do not email submission files to NIST as they will be deleted without being read.
Once you submit a run, you cannot delete it using the submission system. This means you cannot submit a "corrected" version of a run by using the same run tag. The prohibition against remote removal of runs is a safety precaution to ensure no one mistakenly (or deliberately!) overwrites someone else's run. If you need to correct a run, contact NIST with details of the problem. If you need to correct a run on the last night before the submission deadline, submit a new run with a different run tag, and send me mail describing the problem and stating which run the new run should replace.
!!IMPORTANT ACTION TO TAKE NOW: One field in the submission form is a list of organizations that have both applied to participate in TREC 2015 and have submitted the required "Dissemination of TREC Results" form to NIST. The list is sorted by Group ID [the id you selected for your team when you applied]. Right NOW, make sure that you are listed in that field, and that everyone in your group who will be submitting runs recognizes the proper group ID. Contact me if you are not in the list to resolve the issue of why you are not already in it and to get your group inserted into it. Do not wait until right before the deadline to make this check because it may take some time to resolve the issue and the submission deadline is a firm deadline. In particular, do not expect to be accommodated on the night before the deadline if you are not in the list.
Happy TREC-ing,
Ellen
姚应哲
When I open the Temporal Summarization submission page just now ,there are three tasks for us to select , as in the following figure:
I wonder whether there is something wrong with it .
As in the Temporal Summarization 2015 Guidelines ,there are only two tasks : Task 1: Filtering and Summarization ; Task 2: Summarization Only . And in the guideline, you point out that Task 1 ( Filtering and Summarization ) should use TREC-TS-2015F corpus and Task 2( Summarization Only) should use TREC-TS-2015RelOnly corpus .
Now I have done Task 1 ( Filtering and Summarization ) using TREC-TS-2015F corpus , however I could't find the corresponding tasks with the corresponding corpus in the TS submission page to submit my results .
Could you tell me which one should I choose ? Thanks very much!
your YingZhe Yao
Fernando Diaz
YingZhe,
Apologies for the confusion. We wanted to accommodate any teams that worked with the full KBA corpus, instead of only TREC-TS-2015F. Anyone using TREC-TS-2015F would select Task 2. Please let me know if you feel this does not reflect the run you've prepared.
We have uploaded a new version of the guidelines which clarifies these distinctions.
Fernando
Sent: Saturday, August 22, 2015 8:56 PM
To: temporalsummarization
Cc: ellen.v...@gmail.com; tre...@googlegroups.com; Richard McCreadie; Fernando Diaz
Subject: Re: TREC 2015 Temporal Summarization submission system now open
姚应哲
Hi , Fernando,
