KBA track participants,
The TREC runs submission system is now open for KBA track runs (CCR and SSF tasks, only). The submission form is linked to from both
the Tracks page in the Active Participants' part of the TREC web site (i.e.,
http://trec.nist.gov/act_part/tracks14.html)
and the Results Submission page (which in turn is linked from
the main page of the Active Participants' section).
The submission deadline for KBA track runs is September 10. A submission deadline of Sept. 10 effectively means that
you must successfully submit your run prior to NIST staff arriving
at NIST on Sept. 11---basically by 6:00am EDT on Sept. 11. This deadline is a firm deadline.
To submit a run, fill in the submission form by answering
questions that describe the run and specify the file to upload.
After you click submit, the submission system will run a
validation script that will test the submission file for various
kinds of formatting errors. A pointer to this script is on both
the active participants' track page and in the 'Tools' section of the
active participants' web site. Over
the years NIST has found that strict checking of the "sanity" of
an input file leads to far fewer problems down the line as it
catches a lot of mistakes in the run at a time that the submitter
can actually correct them. You are strongly encouraged to use the
script to test your submission file prior to submitting the file
to NIST. Invoke the script giving the run file name as the
argument to the script and a log file will be created. The
log will contain error messages if any errors exist, and will then list the number of suggestions per relevance level for each topic in the run. If any errors are found
by the script at the time the
run is submitted, the submission system will reject the run.
Rejected
runs are not considered to be submitted; indeed, no
information is retained about rejected runs
. If there are no errors found, the system will not return a copy of the log file to the submitter. Thus, you will not see the counts of the number of suggestions at each relevance level in your run unless you use the check script locally on your own machine before submitting the run.
Submitting a
run through the submission system is the only acceptable way to
send a run to NIST. In particular,
do not email
submission files to NIST as they will be deleted without being read.
Once you submit a run, you cannot delete it using the
submission system. This means you cannot submit a "corrected" version
of a run by using the same run tag. The prohibition against remote
removal of runs is a safety precaution to ensure no one mistakenly (or
deliberately!) overwrites someone else's run. If you need to correct a
run, contact NIST with details of the problem. If you need to correct a
run on the last night before the submission deadline, submit a new run
with a different run tag, and send me mail describing the problem and
stating which run the new run should replace.
One field in the submission form is a list of organizations that
have both applied to participate in TREC 2014 and
have submitted the required "Dissemination of TREC Results" form to NIST.
The list is sorted by Group ID [the id you selected for your team when
you applied]. Make
sure you are listed in that field, and contact me if you are
not. You should make that check
now since it could take some time
to resolve the issue of why you are not already in the list and to get
your group inserted into it, and as stated above, the submission deadline is a firm
deadline.
Regards,
Ellen