Same score file, different results

[Teodors Eglītis]

Hello,

I acquire different results using the same score fils, but different tools -- collect_results.py; "bob bio metrics" and "bob.measure.eer".

I am using the stable BOB version:

(base) teo@Teodorss-MacBook-Pro BOB7 % conda activate BOB7

(BOB7) teo@Teodorss-MacBook-Pro BOB7 % conda list

# packages in environment at /Users/teo/opt/anaconda3/envs/BOB7:

#

# Name                    Version                   Build  Channel

blas                      1.0                         mkl    defaults

bob                       7.0.0            py37haafa87e_0    https://www.idiap.ch/software/bob/conda

bob.bio.base              4.1.0            py37h03d05df_0    https://www.idiap.ch/software/bob/conda

bob.blitz                 2.0.20           py37h1769a8a_0    https://www.idiap.ch/software/bob/conda

bob.core                  2.2.4            py37h681806a_0    https://www.idiap.ch/software/bob/conda

bob.extension             4.0.0            py37ha775a6e_0    https://www.idiap.ch/software/bob/conda

bob.io.base               3.0.7            py37hce533d4_0    https://www.idiap.ch/software/bob/conda

bob.io.image              2.4.5            py37he3864fc_0    https://www.idiap.ch/software/bob/conda

bob.io.matlab             2.0.15           py37hf85c1a1_0    https://www.idiap.ch/software/bob/conda

bob.learn.activation      2.0.15           py37h681806a_0    https://www.idiap.ch/software/bob/conda

bob.learn.boosting        2.0.18           py37h635c753_0    https://www.idiap.ch/software/bob/conda

bob.learn.em              2.1.5            py37h877cdbf_0    https://www.idiap.ch/software/bob/conda

bob.learn.linear          2.1.7            py37h635c753_0    https://www.idiap.ch/software/bob/conda

bob.math                  3.1.5            py37hd59a8b7_0    https://www.idiap.ch/software/bob/conda

bob.measure               4.2.0            py37hdc7fa44_0    https://www.idiap.ch/software/bob/conda

<...>

Using "collect_results.py" script:

(BOB7) teo@Teodorss-MacBook-Pro BOB7 % collect_results.py -vv --directory "/Users/teo/RESULTS/test_walking_20_BB_RS_SVM_GS_LOCAL/test_walking_20" -s -c EER

----------------------------------------------------------------------------------------------------

dev  nonorm     dev  ztnorm      eval nonorm    eval ztnorm            directory

----------------------------------------------------------------------------------------------------

28.183%            None            None            None        N4/G2/A1

28.249%            None            None            None        N6/G6/A2

28.249%            None            None            None        N1/G6/A2

28.249%            None            None            None        N4/G6/A2

28.249%            None            None            None        N3/G6/A2

28.249%            None            None            None        N2/G6/A2

28.249%            None            None            None        N5/G6/A2

29.377%            None            None            None        N5/G2/A1

30.106%            None            None            None        N5/G4/A1

Using the best performing parameter combination -- "N4/G2/A1":

(BOB7) teo@Teodorss-MacBook-Pro BOB7 % bob bio metrics "/Users/teo/RESULTS/test_walking_20_BB_RS_SVM_GS_LOCAL/test_walking_20/N4/G2/A1/nonorm/scores-dev" -c eer

[Min. criterion: EER ] Threshold on Development set `/Users/teo/RESULTS/test_walking_20_BB_RS_SVM_GS_LOCAL/test_walking_20/N4/G2/A1/nonorm/scores-dev`: 5.266106e-01

=====================  ===============

..                     Development

=====================  ===============

Failure to Acquire     29.4%

False Match Rate       39.8% (212/532)

False Non Match Rate   39.8% (106/266)

False Accept Rate      28.1%

False Reject Rate      57.6%

Half Total Error Rate  39.8%

=====================  ===============

Doing the same, only this time using a self-made script that reads the score file and produces EER using "bob.measure.eer":

>>> import bob.measure

>>> print(bob.measure.__version__)

Name                EER

/N4/G2/A1         39.849624060150376

<..>

This seems to be in line with the result produced by "bob bio metrics".

Does "bob bio metrics"/bob.measure should be considered more precise than the "collect_results.py" script, maybe I am missing here something?

Best,

Teo

[Amir Mohammadi]

Hi Teo,

I have never seen this collect_results.py  script :)

I will open an issue to see if we can remove it.

Yes bob bio metrics or bob bio evaluate  or bob bio roc  are the scripts to be used for evaluation.

Try running bob bio --help  to see what scripts are available.

Best,

Amir

--

-- You received this message because you are subscribed to the Google Groups bob-devel group. To post to this group, send email to bob-...@googlegroups.com. To unsubscribe from this group, send email to bob-devel+...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/bob-devel or directly the project website at http://idiap.github.com/bob/

---

You received this message because you are subscribed to the Google Groups "bob-devel" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bob-devel+...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/bob-devel/dc74b86e-5bd0-442b-88fb-654105541487%40googlegroups.com.

[Manuel Günther]

Hi Teo, Amir,

I had added that script a while ago to make it easier to compare results when running a parameter search and, thus, having too many results to plot them into one graph.

Theoretically, this script should use the same metrics as `bob bio metrics`, but it might be that the criterion means something else here.

Best regards

Manuel

[Teodors Eglītis]

Amir, Manuel, thank you for the answers!

Teo