Submissions open for Health Misinformation - Deadline 9/2

[Ian Soboroff]

The TREC runs submission system is now open for Health Misinformation Track runs. The submission form is linked to from both the Tracks page in the Active Participants’ part of the TREC web site and the Results Submission page (which in turn is linked from the main page of the Active Participants’ section).The submission deadline for track runs is September 2. A September 2 deadline has now become interpreted as runs must be submitted before NIST staff start work on September 3, so call it 8:00AM EDT on September 3. At that time, the submission system for Health Misinformation track runs will be shut down and it will then be too late to submit a run.To submit a run, fill in the submission form by answering questions that describe the run and specify the file to upload. After you click submit, the submission system will run a validation script that will test the submission file for various kinds of formatting errors. A pointer to this script is on both the active participants’ track page and in the ‘Tools’ section of the active participants’ web site. Over the years NIST has found that strict checking of the “sanity” of an input file leads to far fewer problems down the line as it catches a lot of mistakes in the run at a time that the submitter can actually correct them. You are strongly encouraged to use the script to test your submission file prior to submitting the file to NIST. Invoke the script giving the run file name as an argument to the script and an error log file will be created. The error log will contain error messages if any errors exist, and will say that the run was successfully processed otherwise (note all output is directed to this log file, not to STDOUT). If any errors are found by the script at the time the run is submitted, the submission system will reject the run. Rejected runs are not considered to be submitted; indeed, no information is retained about rejected runs.Submitting a run through the submission system is the only acceptable way to send a run to NIST. Since the metadata collected on the submission form is for a single run only, you must submit your runs one at a time. The results file you upload can be compressed using gzip, but you cannot use archive files such as zip or tar. The results file must contain exactly one complete run: at least one document retrieved for each topic in the test set and no more than 1000 documents retrieved per topic.Runs have an id called the run tag. The tag must be unique across all tracks and all groups. If you use the same tag as a run that has already been submitted, the submission system will tell you that the tag is already in use and you will need to select a new tag (so run ids such as ‘run1’ are a bad idea).  The ‘RUN ID’ given in the submission form and the tag contained within the submission file must match exactly; the submission system will reject the run if they do not. Run tags must be no longer than 15 alphanumeric characters.Once you submit a run, you cannot delete it using the submission system. This means you cannot submit a “corrected” version of a run by using the same run tag. The prohibition against remote removal of runs is a safety precaution to ensure no one mistakenly (or deliberately!) overwrites someone else’s run. If you need to correct a run, contact NIST with details of the problem. If you need to correct a run on the last night before the submission deadline, submit a new run with a different run tag, and send me mail describing the problem and stating which run the new run should replace.IMPORTANT ACTION TO TAKE NOW: One field in the submission form is a list of organizations that have both applied to participate in TREC 2021 and have submitted the required “Dissemination of TREC Results” form to NIST. The list is sorted by Group ID [the id you selected for your team when you applied] where case is significant to the sort. Right now, make sure that you are listed in that field, and that everyone in your group who will be submitting runs recognizes the proper group ID. Contact me if you are not in the list to resolve the issue of why you are not already in it and to get your group inserted into it. Do not wait until right before the deadline to make this check because it may take some time to resolve the issue. In particular, do not expect to be accommodated on the night before the deadline if you are not in the list.

[Gordon V. Cormack]

The script seems to expect the topics to be in the range 1..71, but I think the correct range is 101..150.  Am I missing something?

Also, you need a link under "active participants/submission forms"

[Soboroff, Ian M. (Fed)]

Sorry... I updated the website links right before I shipped the message, but the website hasn't caught up yet.

Ian

--
You received this message because you are subscribed to the Google Groups "TREC Health Misinformation Track" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trec-health-misinforma...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trec-health-misinformation-track/31b04a91-be7d-41ba-9d09-2d333b4faf82n%40googlegroups.com.

[Gordon V. Cormack]

OK, I'm totally confused.  I think I got the wrong script and I don't see the right one.  Maybe the web pages haven't updated yet.

[Gordon V. Cormack]

It's all good now, thanks.  Runs submitted.

[Mark D. Smucker]

If you are having trouble finding the submission link:

https://ir.nist.gov/trecsubmit/misinfo.html

Mark

On Mon, Aug 30, 2021 at 6:40 PM Gordon V. Cormack <corma...@gmail.com> wrote:

It's all good now, thanks.  Runs submitted.

--
You received this message because you are subscribed to the Google Groups "TREC Health Misinformation Track" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trec-health-misinforma...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trec-health-misinformation-track/719be5e8-0eef-4369-842d-0976ec4d576dn%40googlegroups.com.