TREC CAR Y3 submission details, validation code, and code to convert rankings into populated pages

39 views
Skip to first unread message

Laura Dietz

unread,
Jul 9, 2019, 3:30:05 PM7/9/19
to trec...@googlegroups.com
Dear CAR folks,

The validation code for TREC CAR Y3 submission is ready (I hope everything works, let me know if there are any issues)

The same package also includes code to convert a section-level ranking (in TREC's run file format) into a populated page.

Detailed information here: http://trec-car.cs.unh.edu/

August 1, 2019: TREC CAR Y3 Submission deadline (link coming soon!)
July 8: Release of Y3 validation and population code; also see v2.3 data release For the Y3 Submission:
  1. Each outline in benchmarkY3test should be populated with exactly 20 passages from the paragraphCorpus.
  2. Each team can submit up to 10 runs; but for assessment purposes only one can be marked as highest priority and only two marked with medium priority.
  3. See more: “Y3 Submission Format” below and Y3 validation rules
June 20: New v2.3 data release
May 8: Task and guidelines for Y3 evaluation (v2.3)!



Quick instructions:

1. Download the code from here:

https://github.com/TREMA-UNH/car-convert-ranking-to-ordering


2. Download, don't unpack! place as-is in working directory  (or set the path on the command line)

http://trec-car.cs.unh.edu/datareleases/v2.3/paragraph_ids.txt.xz 


3. Download and unpack, then identify the file called benchmarkY3test.cbor-outlines.cbor

http://trec-car.cs.unh.edu/datareleases/v2.3/benchmarkY3test.public.tar.gz


4. Install python requirement with

python3 setup.py install

You can avoid using sudo through the use of anaconda, or you can also install it into user space with this command

python3 setup.py install --user


5. This is the command line to run the validation for one submission file called ${submission}
python3 y3_validate_submission.py --json-file ${submission} --submission-check-y3 --outline-cbor benchmarkY3test.cbor-outlines.cbor



Cheers,
Laura
Reply all
Reply to author
Forward
0 new messages