TREC 2019 CAR track submission system open; you must validate your run file before submitting

[ellen.voorhees]

Dear TREC 2019 CAR track participants,

The TREC runs submission system is now open for CAR track runs.  The submission form is linked to from both the tracks page in the Active Participants' part of the TREC web site and the Results Submission page.  You will need to give the Active Participants' login and password to reach these pages.

The submission deadline for track runs is August 1. A submission deadline of August 1 effectively means that you must successfully submit your run prior to me arriving at NIST on Aug 2---basically by 7:00am EDT on Aug 2.  At that time, I will close the submission system for Precision Medicine track runs.  Once I close the submission system, it will be too late to submit runs.

To submit a run,  fill in the submission form by answering questions that describe the run and specify the file to upload. The metadata collected from the submission form pertains to the single run being submitted.   The file containing the run can be compressed using gzip or bzip2, if desired, but you cannot use archive files such as tar or zip.

Runs have an id called the run tag.  The tag must be unique across all tracks and all groups.  If you use the same tag as a run that has already been submitted, the submission system will tell you that the tag is already in use and you will need to select a new tag (so run ids such as 'run1' are a bad idea).   

Since the deadline is less than two weeks away, I have decided to have the submission system not validate submitted CAR track files. [The submission system is not playing nicely with the validation script largely because of the strict access restrictions engendered by the submission machine living outside the NIST firewall.]  As such, the submission system will accept any file you tell it to upload as your run.  Thus, it is vital that you validate that file yourself before you submit it.  Laura has made a variety of tools available to easily do this checking; see the track web site for details. While the submission system will accept any file given to it, subsequent processing of the file will necessarily assume that it is a correctly formatted file.  It is your responsibility to make sure that that is so. Neither NIST nor the track organizers is going to attempt to salvage a malformed run, so malformed runs will be discarded.

Once you submit a run, you cannot delete it using the submission system.  This means you cannot submit a "corrected" version of a run by using the same run tag.  The prohibition against remote removal of runs is a safety precaution to ensure no one mistakenly (or deliberately!) overwrites someone else's run.  If you need to correct a run, contact NIST with details of the problem.  If you need to correct a run on the last night before the submission deadline, submit a new run with a different run tag, and send me mail describing the problem and stating which run the new run should replace.

IMPORTANT ACTION TO TAKE NOW: One field in the submission form is a list of organizations that have both applied to participate in TREC 2019 and have submitted the required "Dissemination of TREC Results" form to NIST. The list is sorted by Group ID [the id you selected for your team when you applied] where case is significant to the sort. Right now, make sure that you are listed in that field, and that everyone in your group who will be submitting runs recognizes the proper group ID. Contact me if you are not in the list to resolve the issue of why you are not already in it and to get your group inserted into it.  Do not wait until right before the deadline to make this check because it may take some time to resolve the issue.  

Ellen