qrels from last year

[Luca Soldaini]

Hi,

I've obtained the qrels file from last year from the official github repository. However, I'm don't fully understand what's the difference between the 24 files in the qrels/ directory. Which one(s) were used to generate the evaluation results shown in the 2015 proceedings? 

-Luca

[João]

Hi Luca,

Thanks for your question. The filenames have some coding that needs further explanation, I will add a readme file there.

The coding is the following:

<assessment_dimension>.<set_of_assessments>.clef2015.<topic_description>.<graduation>.txt

<assessment_dimension>: either qread or qrels, for readability scores and topical relevance scores

<set_of_assessments>: eng, multi or merged. We used in CLEF2015 only the 'eng' assessments. The 'multi' was later created based only on the multi language runs, and the 'merged' is the merge of the other two.

<topic_description>: test or qtest: it is just the name of the topics, e.g., either 'qtest.1' or just '1'

<graduation>: graded or bin: graded or binary relevance

Finally, answering your question, we used the following files to generate the official evaluation results:

qread.eng.clef2015.test.bin.txt

qread.eng.clef2015.test.graded.txt

qrels.eng.clef2015.test.bin.txt

qrels.eng.clef2015.test.graded.txt

Best,

--
João Palotti

PhD Student

Vienna University of Technology, Austria

Website: joaopalotti.com

Twitter: @joaopalotti

--
You received this message because you are subscribed to the Google Groups "CLEF eHealth - Information retrieval task" group.
To post to this group, send email to clef-eheal...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/clef-ehealth-ir-task/97137b7a-646c-47fb-8dc7-24ea4d20c40d%40googlegroups.com.

[Luca Soldaini]

João,

Thank you for the clarification.

Best,

Luca

[guido]

Hi Luca

to further add to Joao's answer. The script used to generate the evaluation distributed to CLEF participants (the evaluation packs are also in github) is https://github.com/CLEFeHealth/CLEFeHealth2015Task2/blob/master/evaluate.sh (just added: this may have local paths, etc, so it may be not run off-the-shelf, but it gives you pointers about how the evaluation was run). Some details (including the evaluation command) are also in the 2015 task overview paper (section 2.9).

Thanks,

g

To post to this group, send email to clef-ehealth-ir-task@googlegroups.com.

[Luca Soldaini]

Guido, 

Thanks for the extra information. However, I am still struggling with the qrels/query files. For example, for the first query in clef2015.test_with_narratives.queries-EN.txt (“many red marks on legs after traveling from us”, which should correspond to the RMSF disease), the first five relevant results in qrels.eng.clef2015.test.graded.txt are:

1. aldf.1864_12_000027 → page about lyme disease

2. arthr0949_12_000974 → page about vessel inflammation 

3. baby-2032_12_000032 → page about toddler skin rash

4. baby-2032_12_000232 → another page about baby skin rash

5. lupus1314_12_000069 → page about lupus symptoms

Point is that to me none of these pages seem to be relevant at all to the query or to the diagnosis associated with the query.  Am I missing something in how the query should be mapped to qrels? In that case, I’d very happy to know.

Best,

Luca

To post to this group, send email to clef-eheal...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/clef-ehealth-ir-task/97137b7a-646c-47fb-8dc7-24ea4d20c40d%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "CLEF eHealth - Information retrieval task" group.

To post to this group, send email to clef-eheal...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/clef-ehealth-ir-task/d366c6d0-a403-4f75-a52f-8611736db6bc%40googlegroups.com.

[João]

Hi Luca,

Thanks for your very good question again.

As you can see, they are all marked as "Somewhat relevant" (1), not as "Highly relevant" (2).

We admitted a flexible interpretation on the meaning of somewhat relevant, i.e., we left the subjective decision on what is somewhat relevant to the assessors.

A document can be marked as somewhat relevant if, for example, is it on the specific disease, but not very complete, or on some related disease that could be a potential diagnosis.

For this specific case that you mentioned, we collected the following comment from the assessor:

< As the correct diagnosis is not a very common one I couldn't find any relevant documents....some documents mentioned petechia and potential differential diagnoses which I marked as "somewhat relevant". >

Note though, that the criteria of somewhat relevant might be different for another query assessed by another medical student.

Best,

--
João Palotti

PhD Student

Vienna University of Technology, Austria

Website: joaopalotti.com

Twitter: @joaopalotti

To view this discussion on the web visit https://groups.google.com/d/msgid/clef-ehealth-ir-task/CAOT93UOnNfrk%3DAVQW7vqxgnnbtYZP%2BM-X1BAsZvZdmTx8Bu4Wg%40mail.gmail.com.

[Luca Soldaini]

Joao, 

Thanks for the update; that makes sense.

-Luca