Relevance feedback (RF), in an information retrieval search, is the two-part process in which a user judges, either explicitly or implicitly, existing information returned from a search, and the retrieval system uses those judgments and that information to perform a new search, returning more information to the user. Relevance feedback has been one of the successes of information retrieval research for the past 30 years. It has proven to be worthwhile in a wide variety of settings, both when actual user feedback is available, and when the user feedback is implicit.

Despite this success, there has been comparatively few research advances in RF in recent years.  There is no general agreement of what the best RF approach is, or what relative benefits and costs of the various approaches are.  In part, that's because relevance feedback is hard to study, evaluate, and compare.  It is difficult to separate out the effects of an initial retrieval run, the decision procedure to determine what documents will be looked at, the user dependent relevance judgment procedure (including interface), and the actual RF reformulation algorithm. Setting up a framework to look at these separate effects for future research will be an important goal for this track.

The track is being set up as at least a two year effort.  The first year will focus on the second part of the relevance feedback task: taking judgments and the original topic as input and performing a new search.  All participating groups will use exactly the same sets of documents and judgments. The groups will have complete freedom to make use of this relevance information in whatever fashion they wish.  There will be multiple sets of relevance information, which will vary primarily in the amount of relevance information supplied. The groups will run their algorithms and submit a ranked list of documents for each set of relevance information.  These ranked lists will be evaluated by NIST.

The second year will focus more on the first part of the RF task: finding good documents or sources of information for the user to judge.  Exactly what sorts of information will be used will be determined by discussion among the participants.

This page will serve as the central site for the task.  We will add links to Guidelines and other information as they become available. Participants are encouraged to make their interests in this area known during the discussions.

Coordinators:

Chris Buckley (chrisb  - -  sabir.com)

Steve Robertson (ser  - -  microsoft.com)

Please send mail to Chris Buckley to join the mailing list and group.  Note that you still need to apply to the TREC Workshop to become an actual participant.

Relevance Feedback 2008 Links

• Current Guidelines
• Guidelines RF08
• Input relevance information, This is a single tar file containing 4 files ./qrels.RF08.[BCDE]. of size (lines, words, bytes):
• wc qrels.RF08.[BCDE]
      264    1056    6916 qrels.RF08.B
     1584    6336   41478 qrels.RF08.C
     2640   10560   69151 qrels.RF08.D
    54631  218524 1396160 qrels.RF08.E
  
  
• qrels.RF08.tar
• Topics. 2 files, with topics.RF08.man being the 25 topics subset if a group wants to do manual judgments for a run.
  
•   wc topics.RF08*
   264 1020 8006 topics.RF08
    25   98  750 topics.RF08.man
• topics.RF08 
• topics.RF08.man
• Run submission link will be added at a later date.
  

Historical Links for Track:

• Proposal for Relevence Feedback 2008 Track 
• Workshop slides from TREC 2007 Relevance Feedback Workshop
• Summary slides of TREC 2007 Relevance Feedback Workshop