Metrics

[marcosfp 97]

Hi,

First of all, my apologies since I am not sure if this is the appropriate forum to consult this. I am trying to compute some metrics over my rankings using the trec_eval and compatibility measures from TREC 2020 Health Misinfo. However, I have some doubts:

• Is there a way to compute the individual NDCG for correctness? Since when generating the derived-qrels it only outputs a file called binary.useful-correct.
• Is there a way to compute the combined harmful and helpful compatibility measures, as used in the overview paper? Since the gen-derived-qrels only outputs harmful.only and helpful.only files.

I would appreciate any help.

Best,

Marcos

[Mark D. Smucker]

Hi Marcos,

1. There should be a readme that notes "Can only be correct if useful."  The NIST assessors only judged the efficacy in cases of a useful document. 

2. The combined measure was simply compatibility on help minus compatibility minus harm.  Sorry that wasn't clear.

Mark

--
You received this message because you are subscribed to the Google Groups "TREC Health Misinformation Track" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trec-health-misinforma...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trec-health-misinformation-track/5eeb8379-ecd5-43dd-be91-fab582108998n%40googlegroups.com.

[Mark D. Smucker]

That should read: The combined measure was simply compatibility on help minus compatibility on harm.

[marcosfp 97]

Thank you very much for the clarification.

Best,

Marcos