Questions about Final Score in Task 1
57 views
Skip to first unread message
You “Neil” Zhang
May 20, 2024, 2:39:21 AM5/20/24
to SONICOM LAP Challenge
Hi LAP challenge organizers,
I appreciate the efforts to address measurement setup biases among datasets in Task 1. However, I have some concerns regarding the current method for ranking classifier accuracy that I would like to discuss.
The current ranking strategy appears to encourage solutions that achieve lower accuracy. This could potentially lead to strategies that disguise data across datasets rather than genuinely mitigating cross-dataset differences, which could be counterproductive to the task's objective.
I propose considering a ranking method where the closest results to a random guess are ranked higher.
Look forward to your thoughts on this. Thank you for your attention to this matter.
The current ranking strategy appears to encourage solutions that achieve lower accuracy. This could potentially lead to strategies that disguise data across datasets rather than genuinely mitigating cross-dataset differences, which could be counterproductive to the task's objective.
I propose considering a ranking method where the closest results to a random guess are ranked higher.
Look forward to your thoughts on this. Thank you for your attention to this matter.
Best,
Neil
You (Neil) Zhang
Ph.D. Candidate
Audio Information Research Lab
University of Rochester
504 Computer Studies Building,
Rochester, NY, 14627
Aidan Hogg
May 20, 2024, 11:48:12 AM5/20/24
to SONICOM LAP Challenge
Hi Neil,
Thank you for your comment.
Just to clarify, the current evaluation does encourage methods with lower accuracy (i.e. closer to random chance in the classifier). The ranking is the inverse of the accuracy; therefore, the lower the accuracy, the higher the ranking.
In answer to your other question: "This could potentially lead to strategies that disguise data across datasets rather than genuinely mitigating cross-dataset differences". This will be addressed by the Stage 1 validation, which will determine if the HRTF is still realistic and has not been destroyed by the harmonisation process.
On a side note, if you are suggesting a ranking based on a distance to the chance accuracy level instead of 0%, it will not make a difference. That is to say, whether you add, subtract, or multiply 12.5%, it does not matter. The classifier will always have a chance level of 12.5% anyway, as the classifier will always have to select one of the eight datasets as an output, no matter the input. Therefore, it would not be possible to modify an HRTF to get the classifier to perform at a lower accuracy than that, even if you completely destroy the HRTF.
I hope this helps. Please let us know if anything is still unclear.
Cheers,
Aidan
Thank you for your comment.
Just to clarify, the current evaluation does encourage methods with lower accuracy (i.e. closer to random chance in the classifier). The ranking is the inverse of the accuracy; therefore, the lower the accuracy, the higher the ranking.
In answer to your other question: "This could potentially lead to strategies that disguise data across datasets rather than genuinely mitigating cross-dataset differences". This will be addressed by the Stage 1 validation, which will determine if the HRTF is still realistic and has not been destroyed by the harmonisation process.
On a side note, if you are suggesting a ranking based on a distance to the chance accuracy level instead of 0%, it will not make a difference. That is to say, whether you add, subtract, or multiply 12.5%, it does not matter. The classifier will always have a chance level of 12.5% anyway, as the classifier will always have to select one of the eight datasets as an output, no matter the input. Therefore, it would not be possible to modify an HRTF to get the classifier to perform at a lower accuracy than that, even if you completely destroy the HRTF.
I hope this helps. Please let us know if anything is still unclear.
Cheers,
Aidan
---------------------------------------------------------
Dr Aidan Hogg
Lecturer at Queen Mary University of London
Honorary Research Associate at Imperial College London
Centre for Digital Music
Electronic Engineering and Computer Science
Queen Mary University of London
327 Mile End Road, London
E1 4NS, U.K
Email: a.h...@qmul.ac.uk
Personal Website: aidanhogg.uk
QMUL Group Website: c4dm.eecs.qmul.ac.uk
QMUL Website: qmul.ac.uk/eecs/people/profiles/aidanhogg
Imperial Group Website: axdesign.co.uk
Imperial Website: imperial.ac.uk/people/a.hogg
Reply all
Reply to author
Forward
0 new messages