TREC CAR Y3 submission details, validation code, and code to convert rankings into populated pages

Skip to first unread message

Laura Dietz

Jul 9, 2019, 3:30:05 PM7/9/19
Dear CAR folks,

The validation code for TREC CAR Y3 submission is ready (I hope everything works, let me know if there are any issues)

The same package also includes code to convert a section-level ranking (in TREC's run file format) into a populated page.

Detailed information here:

August 1, 2019: TREC CAR Y3 Submission deadline (link coming soon!)
July 8: Release of Y3 validation and population code; also see v2.3 data release For the Y3 Submission:
  1. Each outline in benchmarkY3test should be populated with exactly 20 passages from the paragraphCorpus.
  2. Each team can submit up to 10 runs; but for assessment purposes only one can be marked as highest priority and only two marked with medium priority.
  3. See more: “Y3 Submission Format” below and Y3 validation rules
June 20: New v2.3 data release
May 8: Task and guidelines for Y3 evaluation (v2.3)!

Quick instructions:

1. Download the code from here:

2. Download, don't unpack! place as-is in working directory  (or set the path on the command line) 

3. Download and unpack, then identify the file called benchmarkY3test.cbor-outlines.cbor

4. Install python requirement with

python3 install

You can avoid using sudo through the use of anaconda, or you can also install it into user space with this command

python3 install --user

5. This is the command line to run the validation for one submission file called ${submission}
python3 --json-file ${submission} --submission-check-y3 --outline-cbor benchmarkY3test.cbor-outlines.cbor

Reply all
Reply to author
0 new messages