TREC 2014 Fedrated Web Seach track submissions open
18 views
Skip to first unread message
Ellen Voorhees
Jul 28, 2014, 2:55:06 PM7/28/14
to trec-...@googlegroups.com
Federated Web Search Track participants,
The TREC runs submission system is now open for Federated Web Search track runs. The submission form is linked to from both the Tracks page in the Active Participants' part of the TREC web site (http://trec.nist.gov/act_part/tracks14.html) and the Results Submission page (which in turn is linked from the main page of the Active Participants' section).
The submission deadline for the vertical selection and resource selection tasks is August 18. For the results merging task, the deadline is September 15. A submission deadline of date X effectively means that you must successfully submit your run prior to NIST staff arriving at NIST on date X+1---basically by 7:00am EDT on August 19 or September 16, respectively.
To submit a run, fill in the submission form by answering questions that describe the run and specify the file to upload. After you click submit, the submission system will run a validation script that will test the submission file for various kinds of formatting errors. A pointer to this script (one per task) is on both the active participants' track page and in the 'Tools' section of the active participants' web site. Over the years NIST has found that strict checking of the "sanity" of an input file leads to far fewer problems down the line as it catches a lot of mistakes in the run at a time that the submitter can actually correct them. You are strongly encouraged to use the script to test your submission file prior to submitting the file to NIST. Invoke the script giving the run file name as the argument to the script and an error log file will be created. The error log will contain error messages if any errors exist. If any errors are found by the script at the time the run is submitted, the submission system will reject the run. Rejected runs are not considered to be submitted; indeed, no information is retained about rejected runs. Submitting a run through the submission system is the only acceptable way to send a run to NIST. In particular, do not email submission files to NIST as they will be deleted without being read.
Please note that you can upload compressed files (using gzip or bzip2) as your submission.
Once you submit a run, you cannot delete it using the submission system. This means you cannot submit a "corrected" version of a run by using the same run tag. The prohibition against remote removal of runs is a safety precaution to ensure no one mistakenly (or deliberately!) overwrites someone else's run. If you need to correct a run, contact NIST with details of the problem. If you need to correct a run on the last night before the submission deadline, submit a new run with a different run tag, and send me mail describing the problem and stating which run the new run should replace.
One field in the submission form is a list of organizations that have both applied to participate in TREC 2014 and have submitted the required "Dissemination of TREC Results" form to NIST. The list is sorted by Group ID [the id you selected for your team when you applied]. Make sure you are listed in that field, and contact me if you are not. You should make that check now since it could take some time to resolve the issue of why you are not already in the list and to get your group inserted into it, and the submission deadline is a firm deadline.
Best regards,
Ellen Voorhees
The TREC runs submission system is now open for Federated Web Search track runs. The submission form is linked to from both the Tracks page in the Active Participants' part of the TREC web site (http://trec.nist.gov/act_part/tracks14.html) and the Results Submission page (which in turn is linked from the main page of the Active Participants' section).
The submission deadline for the vertical selection and resource selection tasks is August 18. For the results merging task, the deadline is September 15. A submission deadline of date X effectively means that you must successfully submit your run prior to NIST staff arriving at NIST on date X+1---basically by 7:00am EDT on August 19 or September 16, respectively.
To submit a run, fill in the submission form by answering questions that describe the run and specify the file to upload. After you click submit, the submission system will run a validation script that will test the submission file for various kinds of formatting errors. A pointer to this script (one per task) is on both the active participants' track page and in the 'Tools' section of the active participants' web site. Over the years NIST has found that strict checking of the "sanity" of an input file leads to far fewer problems down the line as it catches a lot of mistakes in the run at a time that the submitter can actually correct them. You are strongly encouraged to use the script to test your submission file prior to submitting the file to NIST. Invoke the script giving the run file name as the argument to the script and an error log file will be created. The error log will contain error messages if any errors exist. If any errors are found by the script at the time the run is submitted, the submission system will reject the run. Rejected runs are not considered to be submitted; indeed, no information is retained about rejected runs. Submitting a run through the submission system is the only acceptable way to send a run to NIST. In particular, do not email submission files to NIST as they will be deleted without being read.
Please note that you can upload compressed files (using gzip or bzip2) as your submission.
Once you submit a run, you cannot delete it using the submission system. This means you cannot submit a "corrected" version of a run by using the same run tag. The prohibition against remote removal of runs is a safety precaution to ensure no one mistakenly (or deliberately!) overwrites someone else's run. If you need to correct a run, contact NIST with details of the problem. If you need to correct a run on the last night before the submission deadline, submit a new run with a different run tag, and send me mail describing the problem and stating which run the new run should replace.
One field in the submission form is a list of organizations that have both applied to participate in TREC 2014 and have submitted the required "Dissemination of TREC Results" form to NIST. The list is sorted by Group ID [the id you selected for your team when you applied]. Make sure you are listed in that field, and contact me if you are not. You should make that check now since it could take some time to resolve the issue of why you are not already in the list and to get your group inserted into it, and the submission deadline is a firm deadline.
Best regards,
Ellen Voorhees
Reply all
Reply to author
Forward
0 new messages