TREC RF: Phase 2 input files ready.
Chris Buckley
to be used for Phase 2 input. You can find it in the Active
Participants section of the TREC website with the RF track info.
http://trec.nist.gov/act_part/tracks/relfdbk/rf09_phase1.tar.gz
As described earlier, it's in the form of a compressed tar ball
with 3 sections:
1. A directory with all 30 phase 1 submissions (19 groups).
Files are in standard TREC results format.
2. A qrels file in standard TREC qrels form giving all judgments
of the Phase 1 submissions. Judgments are 2 for highly
relevant, 1 for relevant and 0 for non-relevant.
3. A file which for each Phase 2 participant gives the 7 or 8
Phase 1 runs that should be used by that participant in their
Phase 2 runs. The file format is lines with pairs of
group_prefix run_name
to indicate that group should run with those judged Phase 1
docs from run_name as their input for their Phase 2 RF runs.
Note that all topic ids are of the form "rf09-#"; some submissions
only used numeric topic ids, but they were changed in the files above.
There is one Phase 2 participant who did not submit in Phase 1 -
I assigned Gord Cormack's Waterloo group the group_prefix of "WAT2"
These 7-8 Phase 1 runs for each participant include all
Phase 1 runs of that participant (0-2 of them), plus at least 5 other
runs, chosen semi-randomly. (All runs were ordered by total
number of relevant docs found, and assigned attempting to ensure
that all participants get a mixture of runs with lots of relevant docs,
and runs with fewer.) Every Phase 1 result file will be run by 5
Phase 2 participants.
People are reminded that these topics, results, and judgments
MAY NOT BE USED for any TREC 2009 purpose other than Phase 2
submissions to the RF track. In particular, you absolutely cannot do any
training on your general ClueWeb retrieval algorithm based on this
data, whether it be for the RF track, or for other TREC 2009 tracks.
Let me know of any problems or questions you have about this data
as soon as possible. Thanks
Chris