Learning Curve based evaluation

[Parthasarathi Mukhopadhyay]

I am doing a small experiment with the learning curve script (slightly modified to include F1@5 at limit 1 [F1@1] and F1@5 at limit 3 [F1@3].

The result at different data points:

Finally it says: 

real 0m50.729s
user 3m55.653s
sys 0m8.626s

Final model trained with limit 10000 achieving best NDCG score 0.9161am trying 

Then I'm trying to cross check the scores from the finally trained dataset but found mismatches:

annif eval sdg-nn /home/psm/Documents/ICSSR-Final/ICSSR-2024/Data_Collected/002_FinalDatasets/test-gold.tsv --limit 1 --metric F1@5 --verbosity INFO
2026-02-09T15:16:25.322Z INFO  [omikuji::model] Loading model from data/projects/sdg-omikujiB/omikuji-model...
2026-02-09T15:16:25.322Z INFO  [omikuji::model] Loading model settings from data/projects/sdg-omikujiB/omikuji-model/settings.json...
2026-02-09T15:16:25.322Z INFO  [omikuji::model] Loaded model settings Settings { n_features: 356553, classifier_loss_type: Hinge }...
2026-02-09T15:16:25.322Z INFO  [omikuji::model] Loading tree from data/projects/sdg-omikujiB/omikuji-model/tree0.cbor...
2026-02-09T15:16:25.334Z INFO  [omikuji::model] Loading tree from data/projects/sdg-omikujiB/omikuji-model/tree1.cbor...
2026-02-09T15:16:25.346Z INFO  [omikuji::model] Loading tree from data/projects/sdg-omikujiB/omikuji-model/tree2.cbor...
2026-02-09T15:16:25.358Z INFO  [omikuji::model] Loaded model with 3 trees; it took 0.04s
F1@5:                         0.8271
Documents evaluated:           17493

annif eval sdg-nn /home/psm/Documents/ICSSR-Final/ICSSR-2024/Data_Collected/002_FinalDatasets/test-gold.tsv --limit 3 --metric F1@5 --verbosity INFO
2026-02-09T15:18:06.905Z INFO  [omikuji::model] Loading model from data/projects/sdg-omikujiB/omikuji-model...
2026-02-09T15:18:06.905Z INFO  [omikuji::model] Loading model settings from data/projects/sdg-omikujiB/omikuji-model/settings.json...
2026-02-09T15:18:06.905Z INFO  [omikuji::model] Loaded model settings Settings { n_features: 356553, classifier_loss_type: Hinge }...
2026-02-09T15:18:06.905Z INFO  [omikuji::model] Loading tree from data/projects/sdg-omikujiB/omikuji-model/tree0.cbor...
2026-02-09T15:18:06.917Z INFO  [omikuji::model] Loading tree from data/projects/sdg-omikujiB/omikuji-model/tree1.cbor...
2026-02-09T15:18:06.929Z INFO  [omikuji::model] Loading tree from data/projects/sdg-omikujiB/omikuji-model/tree2.cbor...
2026-02-09T15:18:06.942Z INFO  [omikuji::model] Loaded model with 3 trees; it took 0.04s
F1@5:                         0.6065
Documents evaluated:           17493

annif eval sdg-nn /home/psm/Documents/ICSSR-Final/ICSSR-2024/Data_Collected/002_FinalDatasets/test-gold.tsv --metric NDCG --verbosity INFO
2026-02-09T15:20:05.012Z INFO  [omikuji::model] Loading model from data/projects/sdg-omikujiB/omikuji-model...
2026-02-09T15:20:05.012Z INFO  [omikuji::model] Loading model settings from data/projects/sdg-omikujiB/omikuji-model/settings.json...
2026-02-09T15:20:05.012Z INFO  [omikuji::model] Loaded model settings Settings { n_features: 356553, classifier_loss_type: Hinge }...
2026-02-09T15:20:05.012Z INFO  [omikuji::model] Loading tree from data/projects/sdg-omikujiB/omikuji-model/tree0.cbor...
2026-02-09T15:20:05.024Z INFO  [omikuji::model] Loading tree from data/projects/sdg-omikujiB/omikuji-model/tree1.cbor...
2026-02-09T15:20:05.036Z INFO  [omikuji::model] Loading tree from data/projects/sdg-omikujiB/omikuji-model/tree2.cbor...
2026-02-09T15:20:05.051Z INFO  [omikuji::model] Loaded model with 3 trees; it took 0.04s
NDCG:                         0.9147
Documents evaluated:           17493

Interestingly, it is happening only with the NN backend, not other regular backends that I've deployed (SVC, Omikuji Bonsai, FastText).

Am I making any mistakes here?

Thanks and regards 

Parthasarathi Mukhopadhyay

Professor, Department of Library and Information Science,

University of Kalyani, Kalyani - 741 235 (WB), India

https://orcid.org/0000-0003-0717-9413