TREC 2019 Decisions track runs submission system now open
Ellen Voorhees
The TREC runs submission system is now open for Decisions track runs. The submission form is linked to from both the tracks page in the Active Participants' part of the TREC web site and the Results Submission page. You will need to give the Active Participants' login and password to reach these pages, and again to reach the submission form itself.
The submission deadline for track runs is August 27. A deadline of August 27 officially means runs must be submitted by midnight EDT on August 27, but in practice it means that runs must be submitted before I arrive at NIST on August 28---basically by 7:00am EDT on Aug 28. At that time, I will close the submission system for Decisions track runs. Once I close the submission system, it will be too late to submit runs. Since the deadline is officially Aug 28, I am generally not sympathetic to those who are still submitting runs as I shut down the system.
To submit a run, fill in the submission form by answering questions that describe the run and specify the file to upload. The metadata collected from the submission form pertains to the single run being submitted. A run consists of at least one document retrieved for each topic in the test set. You may have different numbers of documents retrieved for different topics (up to the maximum of 1000 documents) but there must be at least one document retrieved for each topic. The file containing the run can be compressed using gzip or bzip2, if desired, but you cannot use archive files such as tar or zip.
Runs have an id called the run tag. The tag must be unique across all tracks and all groups. If you use the same tag as a run that has already been submitted, the submission system will tell you that the tag is already in use and you will need to select a new tag (so run ids such as 'run1' are a bad idea). The 'RUN ID' given in the submission form and the tag contained within the submission file must match exactly; the submission system will reject the run if they do not. Run tags must be no longer than 15 alphanumeric characters (plus '_' and ‘-‘ and ‘.').
After you click submit, the submission system will run a validation script that will test the submission file for various kinds of formatting errors. There is a link to the script from both the tracks page and the ‘Tools' section of the active participants' web site. Over the years NIST has found that strict checking of the "sanity" of an input file leads to far fewer problems down the line as it catches a lot of mistakes in the run at a time that the submitter can actually correct them. You are very strongly encouraged to use the script to test your submission file prior to submitting the file to NIST. If any errors are found by the script at the time the run is submitted, the submission system will reject the run. Rejected runs are not considered to be submitted; indeed, no information is retained about rejected runs. Submitting a run through the submission system is the only acceptable way to send a run to NIST. In particular, do not email runs. [Seriously, the only thing I will do with an emailed run is discard it, so don’t waste your time and mine by emailing run files to NIST.]
Invoke the script giving the run file name as the argument to the script and an error log file will be created. The error log will contain error messages if any errors exist, and will say that the run was successfully processed otherwise. All output is directed to this log file, nothing to STDOUT.
Once you submit a run, you cannot delete it using the submission system. This means you cannot submit a "corrected" version of a run by using the same run tag. The prohibition against remote removal of runs is a safety precaution to ensure no one mistakenly (or deliberately!) overwrites someone else's run. If you need to correct a run, contact NIST with details of the problem. If you need to correct a run on the last night before the submission deadline, submit a new run with a different run tag, and send me mail describing the problem and stating which run the new run should replace.
IMPORTANT ACTION TO TAKE NOW: One field in the submission form is a list of organizations that have both applied to participate in TREC 2019 and have submitted the required "Dissemination of TREC Results" form to NIST. The list is sorted by Group ID [the id you selected for your team when you applied] where case is significant to the sort. Right now, make sure that you are listed in that field, and that everyone in your group who will be submitting runs recognizes the proper group ID. Contact me if you are not in the list to resolve the issue of why you are not already in it and to get your group inserted into it. Do not wait until right before the deadline to make this check because it may take some time to resolve the issue. In particular, do not expect to be accommodated the night of the deadline if you are not in the list, and do not expect that you will be able to submit a run late because you did not discover that you were not in the list in time.
Ellen