TREC RF: plans for Phase 2
Chris Buckley
finished up at NIST and should be done in a couple of days.
If there are groups who did NOT participate in Phase 1, but who
wish to participate in Phase 2, you need to let me know by this
Wednesday (7/15), so you can be included in the distribution
of Phase 1 results. You should include evidence that you will
be able to complete Phase 2. At this date that means you should
have already indexed at least the B subset of ClueWeb, or have
some pretty strong reasons for believing that indexing it is not
a problem.
The overall plan is put up a tar-ball for Phase 2 on the NIST site in
the participants only section. It will contain
1. All phase 1 submissions.
2. Qrels giving all judgments of the Phase 1 submissions.
3. A file which for each Phase 2 participant gives the 7 or so
Phase 1 runs that should be used by that participant in their
Phase 2 runs.
These 7 (or so) Phase 1 runs for each participant will include all
Phase 1 runs of that participant (0-2 of them), plus at least 5 other
runs, chosen semi-randomly. (I'll order all runs by total number of
relevant docs found, and ensure that all participants get a mixture
of runs with lots of relevant docs, and runs with fewer.) The goal is
to have all Phase 1 runs being run by at least 5 Phase 2 participants.
We hope to have the Phase 2 tar-ball available by this Friday 7/17.
People are reminded that these topics, results, and judgments
MAY NOT BE USED for any TREC 2009 purpose other than Phase 2
submissions to the RF track. In particular, you absolutely cannot do any
training on your general ClueWeb retrieval algorithm based on this
data, whether it be for the RF track, or for other TREC 2009 tracks.
Chris