Guidelines for TREC-IS 2021-B

[Cody Buntain]

Afternoon, all!

We're excited to share details for TREC-IS 2021-B and how we will run the official TREC-IS results in November!

TL;DR – We will use each team's highest-performing system submitted to the leaderboard or to 2021-A for official TREC-IS evaluation in 2021-B. Teams' systems submitted on or before 18 October 2021 will be included.

The main changes we have implemented for 2021-B are two-fold: 

1. First, we are going to use submissions to the GitHub leaderboard repository rather than taking submissions via the TREC form (if you want your submission to be anonymous, we have made allowances for that). 
2. Second, we will use the event data already released for 2021-A for evaluation rather than releasing a new event set. Since the 2021-A set included more events than we used for evaluation, we are generating new assessments for a subset of events that we did not evaluate in 2021-A and going deeper on a few existing events. 

This second point has a major implication in that we can evaluate any system that has submitted to any 2021 evaluation. To this end, we anticipate taking each team's highest performing system as determined by the extant leaderboard, and we will evaluate that system using the 2021-B assessments for the official TREC-IS results.

For more detailed guidance, the 2021-B track guidelines are available on the GitHub repository https://github.com/trecis/trecis.github.io and here. 

If you have any questions, feel free to reach out!

-All the best,

Cody, Richard, and Ian