TREC RF: final guidelines with minor changes, submission info

[Chris Buckley]

RF folks:

I've made minor modifications to the guidelines, and uploaded them
to the Google group as a file.
http://trec-relfeed.googlegroups.com/web/Guidelines09-final

Changes and clarifications to note:
1. only one set of runs per group in Phase 2. There was not a
big demand for more (only one group expressed a preference.) As it is,
we should be submitting a total of 170 runs into the pools. With a
Phase 2 judging budget of only 200-500 docs per topic, that's already
not many new docs per run!

2. Minor corrections taking into account how Phase 1 results were
distributed (as result files instead of qrels files, since qrels
files would have no ordering of their 5 docs, and some groups may
treat the top relevant doc differently from the 5th relevant doc.)

3. Along with that, the naming nomenclature of the Phase 2 submissions
has changed to
<run_basename>.<results_file>
where run_basename may be a max of 6 characters long.
Thus a set of runs from Sabir Research might be named
SAB9RF.base
SAB9RF.SAB.1
SAB9RF.SAB.2
SAB9RF.ABCD.2
SAB9RF.DEF.1
SAB9RF.GHI.1
SAB9RF.JKLM.2
SAB9RF.MNO.2

4. The submission system should be up and running at NIST now,
thanks to Ian. http://ir.nist.gov/trecsubmit/relfdbk.html
The submission system enforces the naming nomenclature - you
can only submit your assigned runs.
At the moment, the "base" run cannot be submitted - that will be
fixed once Ian comes back from vacation next week (let me know if
you need to submit the base run before then).

5. The Phase 2 submission date remains at August 24. That is a
very firm date; the assessing schedule doesn't offer any leeway.

Chris

[Xiaohui(Daniel) Tao]

Hi, Chris

Just to clarify that we need to submit one run according to one relevance
feedback submission (phase 1) assigned to us, is it correct? In this case,
if we have 8 groups' phase 1 submissions (including ours) assigned to us, we
need to submit a set of 8 runs in phase 2?

Thanks for your time and look forward to your reply.

Best regards

Daniel
=========================================
Xiaohui(Daniel) Tao
Research Associate
SIT, Queensland University of Technology
Brisbane QLD 4000 Australia

[Chris Buckley]

On Wed, Aug 5, 2009 at 1:29 AM, Xiaohui(Daniel)
Tao<dannybo...@gmail.com> wrote:
>
> Hi, Chris
>
> Just to clarify that we need to submit one run according to one relevance
> feedback submission (phase 1) assigned to us, is it correct? In this case,
> if we have 8 groups' phase 1 submissions (including ours) assigned to us, we
> need to submit a set of 8 runs in phase 2?

Yes (or quite close). If you have 8 groups assigned to you, you should
submit a set of 9 runs in Phase 2 - one run using each of the
Phase 1 resullts, and a base case run with no explicit relevance feedback.

You can use either the B-subset (same collection as in Phase 1) or the
full English collection for your runs, but please use the same collection
for ALL of your run set - don't mix.

I'll ask Ian to add a question about which collection you used to the
submission form - if you submit runs before he gets back from vacation,
could you note 'B-subset' or 'full-collection' within the short description
of run field on the submission form? Thanks.

Chris