Release of test set, role of expert annotations, test overfitting

[R Quintino]

Hey everyone and organization team,

Based on the facts that the test set is released, and also it's unclear the role/possible use of expert annotations both on train and test set. 

byw are expert annotations allowed? Any limitations apply?

Concern is mostly, assuming some HER2 correlation with visible morphological features at least exists, ex: cancer sub type, an expert pathologist could help annotate only test images (ex: risk/probabilities), if a model is needed, a model could be overfitted to this labels, then saved. Then just use this model predictions as submissions, also pack the overfitted model/code. 

How can this be prevented? Imagining that some rules or submission requirements would prevent cases like this but not clear

thx!

Rui

[Eduardo Conde-Sousa]

Hi Rui

Regarding experts, if they are part of the team, of course you ca use their expertise, but keep in mind to use that expertise/annotations just during the training, not on the test dataset. The code should receive any whole slide and export results without any previous annotations, so if you fell like your code should first find a specific region of the slide, that identification should be encoded somehow, not dependent of any manual annotation.

Regarding correlations, that is a question that I, as organizer, can't answer, but of course if any one else wants, it's fine (sharing ideas is the main purpose of the forum).

All the best

Eduardo

******************************************************************************************

Eduardo Conde-Sousa

INEB – Instituto Nacional de Engenharia Biomédica

i3S – Instituto de Investigação e Inovação em Saúde, Universidade do Porto
Rua Alfredo Allen, 208 | 4200-135 Porto, Portugal

Telf: +351 220408815
www.bimage.ineb.up.pt

http://www.ineb.up.pt/

https://www.i3s.up.pt/scientific-platform?v=91
www.i3s.up.pt

******************************************************************************************

--
You received this message because you are subscribed to the Google Groups "HEROHE Grand-Challenge" group.
To unsubscribe from this group and stop receiving emails from it, send an email to herohe-grand-chal...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/herohe-grand-challenge/7de6b180-6abe-4276-9ff0-ff80bfcdd219%40googlegroups.com.

[R Quintino]

Hi Eduardo, thanks for the reply & clarifications

Think it's more clear for me way the purpose of the submitted code and expert annotations.

In practice, assuming we're all under kind of "code of honor" agreement that's what the submission should do :)  - Generalize on unseen dataset, *without prev knowledge of test images*

as the test dataset is released for the competition eval, overfitting a specific trained model is still a theoretical possibility if someone can at least hint on labels. 

(prob only way to detect this would be to eval on a truly unseen -to parcipants- dataset )

thanks!

To unsubscribe from this group and stop receiving emails from it, send an email to herohe-grand-challenge+unsub...@googlegroups.com.