FedWeb14 relevance judgements

[Ilya Markov]

Dear organizers,

Where can I find relevance judgements for the FedWeb14 tasks?

Thank you,

Ilya.

[Garrick Sherman]

I'd like to second this request. I'm interested in doing work with the FedWeb dataset that requires qrels from other parts of the track than the one I participated in (which means I need more than just what was emailed with my results). Any chance the organizers could post all relevance judgments to the TREC website or to the FedWeb site?

[Thomas Demeester]

Dear all, 

we released the (raw) FedWeb14 relevance judgments.

The file singleJudgmentsFW14_50official.txt contains the complete (34,003) page judgments for the 50 official topics, and singleJudgmentsFW14_10extra.txt contains the (6,451) jugments for the 10 additional topics used in the online evaluation system during the TREC preparation phase. Both files can be found on the TREC 2014 website.

The parameters included in these files are:

• snippetID: the page ID
• userID: the anonymized assessor ID
• pagelabel: the assigned relevance level
• pageproblem: whether the assessor had a problem with visualizing the original page 
  (0 = no problem, confident in the assigned label; 1 = potential problem, less confident)
• watchedvideo: whether the assessor effectively watched (part of) the video contained in the original page (0 = did not watch, and 1 = watched) 

Only snippetID and pagelabel were used for the FedWeb14 evaluation, but the others might be useful for further analysis.

 

If you have any further questions regarding these datafiles, please ask!

Best regards,

Thomas

Op zaterdag 15 november 2014 11:28:41 UTC+1 schreef Ilya Markov:

[Garrick Sherman]

Thanks for posting these judgments!

I'm sure I'm overlooking something obvious, but can you explain why some resources don't seem to have been judged for some topics? For example, for topic 7015, resources e003 and e006 (among several others) don't appear to have been judged at all, but I'm having trouble finding a rationale in the overview paper. Is this just a question of not receiving any results from these resources for these topics, or am I misunderstanding something else?

Thanks for the help!

[Thomas Demeester]

Hi Garrick,

you're correct: we just didn't receive any results from these resources for that particular query.

Best regards,

Thomas

Op donderdag 11 december 2014 06:44:24 UTC+1 schreef Garrick Sherman: