Expressivity of decision rules for FPR/FNR estimation

[Joël Tang]

Hello,

I am having a look at the technical document describing metrics, and I'd like to clarify my understanding of the "Computing per-example ε’s" section.

For a sample to be forgotten s, R^s and U^s are estimated by inference of N retrained and N forgetting models respectively. This sampling is with respect to the randomness issued from initialization, batches composition, order etc.

Then, a function f(.) flattens such outputs into a scalar. Finally, decision rules (that we could call classifiers) try to distinguish samples from R^s vs U^s. Figure 1 shows a very clear example of linear classifier based on the resulting 1-D space (from f(.)). In this case, FPR and FNR can be directly derived (when the classifier is applied on R^s union U^s).

Is it correct to assume that such decision rules are voluntarily made "simple", working on the 1-D output space of f(.), hence preventing overfitting? I could see why it could pose a problem if we didn't have a validation set (from the point of view of the decision rule) and still need to derive TPR and FNR.

Also, the problem of checking whether two sampled populations are drawn from the same distribution reminds me of the purpose of some statistical tests, for instance Kolgomorov-Smirnov. Although we would lose the derivation that is to estimate epsilon, do you think it could make sense, at least having some monotonous behavior w.r.t. it?

Best regards,
Joel Tang