TREC 2018 submission system now open for CAR track runs

[ellen.voorhees]

CAR Track participants,

The TREC runs submission system is now open for Complex Answer Retrieval (CAR) track runs for 2018. The submission form is linked to from both the Tracks page in the Active Participants' part of the TREC web site and the Results Submission page (which in turn is linked from the main page of the Active Participants' section).  You will need to give the active participants' access sequence to reach the form.

The submission deadline for CAR track runs is August 15. A submission deadline of August 15 effectively means that you must successfully submit your run prior to NIST staff turning off the submission system at approximately 7:00am EDT on Aug 16.   This is an absolutely firm deadline so preparations can be made in time for the TREC assessors who start on the morning of Monday, August 20.  

To submit a run, fill in the submission form by answering questions that describe the run and specify the file to upload. After you click submit, the submission system will run a validation script that will test the submission file for various kinds of formatting errors. A pointer to this script is on both the active participants' track page and in the 'Tools' section of the active participants' web site. Over the years NIST has found that strict checking of the "sanity" of an input file leads to far fewer problems down the line as it catches a lot of mistakes in the run at a time that the submitter can actually correct them. You are strongly encouraged to use the script to test your submission file prior to submitting the file to NIST. If any errors are found by the script at the time the run is submitted, the submission system will reject the run. Rejected runs are not considered to be submitted; indeed, no information is retained about rejected runs. Submitting a run through the submission system is the only acceptable way to send a run to NIST.  In particular, do not email runs to NIST.

The metadata collected from the submission form pertains to the single run being submitted.  A run consists of at least one entry (passage id or entity id) for each topic in the test set. You may have different numbers of entries for different topics. The file containing the run can be compressed using gzip or bzip2, if desired, but you cannot use archive files such as tar or zip.

Runs have an id called the run tag.  The tag must be unique across all tracks and all groups.  If you use the same tag as a run that has already been submitted, the submission system will tell you that the tag is already in use and you will need to select a new tag (so run ids such as 'run1' are a bad idea).   The 'RUN ID' given in the submission form and the tag contained within the submission file must match exactly; the submission system will reject the run if they do not.  Run tags must be no longer than 15 characters and can contain only letters, digits, and '_', '-', and '.'.

Once you submit a run, you cannot delete it using the submission system.  This means you cannot submit a "corrected" version of a run by using the same run tag.  The prohibition against remote removal of runs is a safety precaution to ensure no one mistakenly (or deliberately!) overwrites someone else's run.  If you need to correct a run, contact me with details of the problem.  If you need to correct a run on the last night before the submission deadline, submit a new run with a different run tag, and send me mail describing the problem and stating which run the new run should replace.

IMPORTANT ACTION TO TAKE NOW: You should take a look at the submission form NOW so that you know what information you will need to submit.  An especially important part of the form is a list of organizations that have both applied to participate in TREC 2018 and have submitted the required "Dissemination of TREC Results" form to NIST. The list is sorted by Group ID [the id you selected for your team when you applied] where case is significant to the sort. Right now, make sure that you are listed in that field, and that everyone in your group who will be submitting runs recognizes the proper group ID. Contact me if you are not in the list to resolve the issue of why you are not already in it and to get your group inserted into it. Do not wait until right before the deadline to make this check because it may take some time to resolve the issue and the submission deadline is a firm deadline.  In particular, do not expect to be accommodated on the night of the deadline if you are not in the list (nor will you be allowed to submit runs late because you weren't listed).

Ellen

[xuesiy...@gmail.com]

Dear Ellen，

I am really sorry to ask what should I do if Im not clear about the account name and password.

I checked the email when I registered at March, but there are only some information about group name or Id.

So, is there any possible to get account back?

Sincerely

Siyuan

[Zhengyang SONG]

Hello, 

I want to ask whether there is any chance that I can participate in this activity now?

Since I only begin to know this recently and the registration process has ended.

Bests,

Zhengyang

--
You received this message because you are subscribed to the Google Groups "TREC Car" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trec-car+u...@googlegroups.com.
To post to this group, send email to trec...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trec-car/76adcfde-194e-4841-903f-31142391575c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

SONG Zhengyang
IIIS, Tsinghua University '16
Beijing, 100084

P.R.China
Tel:+86 188-1136-8537