TREC 2014 KBA track submission system now open

[Ellen]

KBA track participants,

The TREC runs submission system is now open for KBA track runs (CCR and SSF tasks, only).  The submission form is linked to from both the Tracks page in the Active Participants' part of the TREC web site (i.e., http://trec.nist.gov/act_part/tracks14.html) and the Results Submission page (which in turn is linked from the main page of the Active Participants' section).

The submission deadline for  KBA track runs is September 10.  A submission deadline of Sept. 10 effectively means that you must successfully submit your run prior to NIST staff arriving at NIST on Sept. 11---basically by 6:00am EDT on Sept. 11.   This deadline is a firm deadline.

To submit a run,  fill in the submission form by answering questions that describe the run and specify the file to upload.  After you click submit, the submission system will run a validation script that will test the submission file for various kinds of formatting errors.  A pointer to this script is on both the active participants' track page and in the 'Tools' section of the active participants' web site.  Over the years NIST has found that strict checking of the "sanity" of an input file leads to far fewer problems down the line as it catches a lot of mistakes in the run at a time that the submitter can actually correct them.  You are strongly encouraged to use the script to test your submission file prior to submitting the file to NIST.  Invoke the script giving the run file name as the argument to the script and a log file will be created.  The log will contain error messages if any errors exist, and will then list the number of suggestions per relevance level for each topic in the run.   If any errors are found by the script at the time the run is submitted, the submission system will reject the run.  Rejected runs are not considered to be submitted; indeed, no information is retained about rejected runs.  If there are no errors found, the system will not return a copy of the log file to the submitter.  Thus, you will not see the counts of the number of suggestions at each relevance level in your run unless you use the check script locally on your own machine before submitting the run.

Submitting a run through the submission system is the only acceptable way to send a run to NIST.  In particular, do not email submission files to NIST as they will be deleted without being read.

Once you submit a run, you cannot delete it using the submission system.  This means you cannot submit a "corrected" version of a run by using the same run tag.  The prohibition against remote removal of runs is a safety precaution to ensure no one mistakenly (or deliberately!) overwrites someone else's run.  If you need to correct a run, contact NIST with details of the problem.  If you need to correct a run on the last night before the submission deadline, submit a new run with a different run tag, and send me mail describing the problem and stating which run the new run should replace.

One field in the submission form is a list of organizations that have both applied to participate in TREC 2014 and have submitted the required "Dissemination of TREC Results" form to NIST.  The list is sorted by Group ID [the id you selected for your team when you applied]. Make sure you are listed in that field, and contact me if you are not.  You should make that check now since it could take some time to resolve the issue of why you are not already in the list and to get your group inserted into it,  and as stated above, the submission deadline is a firm deadline.


Regards,

Ellen