Test queries for TREC CAR 2018 (year 2) released!

[Laura Dietz]

Dear TREC CAR participants,

The official evaluation queries are released! The submission will open in a day or two. Submission deadline is August 15

See http://trec-car.cs.unh.edu/datareleases/ for details.

• New test queries in benchmarkY2test! (These are in part from 2018 Wikipedia pages and from AI2's TQA dataset)
  
• The paragraph corpus is the same as v2.0 (released in January). 
  
• No additional training data will be released. Please use train, test200, benchmarkY1train, benchmarkY1test for training.
  
• If you are using the unprocessedAllButBenchmark data for your submission, *you must download the v2.1* version. You are not permitted you to use the v2.0 unprocessedAllButBenchmark dataset for your 2018 submission.
  

Please make sure you are using the latest trec-car-tools versions for java (version 10!) and python. You may get some warnings but it should do the job.

As always: If there is anything that could be a mistake, please send me an email or open a ticket [1].

[1] https://github.com/TREMA-UNH/trec-car-tools/issues

Best,
Laura

[Laura Dietz]

Dear all,

Of course there was a last oopsie in the dataset. (One page had two sections with the same name -- who would have thought?!)

Its fixed now. If you already downloaded the new queries, please download again. The fixed version is v2.1.1.


I also put the validation script online -- make sure you also grab the car.topics file!

Best,
Laura