Test queries for TREC CAR 2018 (year 2) released!

53 views
Skip to first unread message

Laura Dietz

unread,
Jul 25, 2018, 10:14:06 AM7/25/18
to trec...@googlegroups.com

Dear TREC CAR participants,


The official evaluation queries are released! The submission will open in a day or two. Submission deadline is August 15

See http://trec-car.cs.unh.edu/datareleases/ for details.


  • New test queries in benchmarkY2test! (These are in part from 2018 Wikipedia pages and from AI2's TQA dataset)
  • The paragraph corpus is the same as v2.0 (released in January). 
  • No additional training data will be released. Please use train, test200, benchmarkY1train, benchmarkY1test for training.
  • If you are using the unprocessedAllButBenchmark data for your submission, *you must download the v2.1* version. You are not permitted you to use the v2.0 unprocessedAllButBenchmark dataset for your 2018 submission.


Please make sure you are using the latest trec-car-tools versions for java (version 10!) and python. You may get some warnings but it should do the job.


As always: If there is anything that could be a mistake, please send me an email or open a ticket [1].

[1] https://github.com/TREMA-UNH/trec-car-tools/issues


Best,
Laura



Laura Dietz

unread,
Jul 25, 2018, 7:11:08 PM7/25/18
to trec...@googlegroups.com
Dear all,

Of course there was a last oopsie in the dataset. (One page had two sections with the same name -- who would have thought?!)

Its fixed now. If you already downloaded the new queries, please download again. The fixed version is v2.1.1.


I also put the validation script online -- make sure you also grab the car.topics file!

Best,
Laura
Reply all
Reply to author
Forward
0 new messages