On Evaluation Framework

[Andrew Shen]

Are the evaluation methods only going to assume access to the model prediction confidence? In the challenge announcement, it states that "For evaluating forgetting, we will use tools inspired by MIAs, such as LiRA". LiRA and the loss based method in the notebook only uses the predicted confidences of the model, I was wondering if this is going to be the case for the challenge evaluation as well. Or should we expect that the evaluation methods might look at the activations values for different layers of the model or look at the model weights.

Thanks,
Andrew

[Andrew Shen]

I have a follow up question about evaluation, are we considering:
(1): a low false positive setting like LiRA, where we are testing the ability of an adversarial to identify data in the forget set with low false positive rate
(2): or are we considering the average case, such as the overall accuracy (as used in the starter kit notebook) or ROC-AUC of an adversarial trying to identify data in the forget set

There is a likely a tradeoff between performance in the low FP setting and average performance when selecting unlearning algorithms, so it would be helpful if we can know which setting to focus on.

Thanks!
Andrew

[Fabian Pedregosa]

Hi Andrew,

Upon launching the competition, we will release a document with a more detailed description of the evaluation procedure (see also the FAQ which we just updated https://unlearning-challenge.github.io/faq/)

Kind regards

--
You received this message because you are subscribed to the Google Groups "unlearning-challenge" group.
To unsubscribe from this group and stop receiving emails from it, send an email to unlearning-chall...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/unlearning-challenge/2eb3aed7-17bf-414b-90ca-ac6c7bd6c0a4n%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.