Format of score files

[Maria Garcia]

What is the format of the score files?  I cannot find any documentation of what a score file looks like and what each field is supposed to represent?  

I also can't seem to produce any output that is readable.  So far the output is all in HDF5.

When I run the out of the box voxforge example and run the evaluate example I get this error:

(If I don't put * on the -e/-d, it fails and says

$ ~/bobspear117/bob.spear-1.1.7/bin/evaluate.py -e eval/ -d dev/  -c EER -D DET.pdf -x
Traceback (most recent call last):
  File "/home/user/bobspear117/bob.spear-1.1.7/bin/evaluate.py", line 21, in <module>
    sys.exit(facereclib.script.evaluate.main())
  File "/usr/lib/python2.7/site-packages/facereclib-1.2.3-py2.7.egg/facereclib/script/evaluate.py", line 168, in main
    scores_dev = [score_parser(os.path.join(args.directory, f)) for f in args.dev_files]
  File "/usr/local/lib64/python2.7/site-packages/bob/measure/load.py", line 58, in split_four_column
    scores_list = four_column(filename)
  File "/usr/local/lib64/python2.7/site-packages/bob/measure/load.py", line 29, in four_column
    for i, l in enumerate(open(filename, 'rt')):
IOError: [Errno 21] Is a directory: './dev/'

I'm using Centos 7

$ ~/bobspear117/bob.spear-1.1.7/bin/evaluate.py -e eval/* -d dev/*  -c EER -D DET.pdf -x
Traceback (most recent call last):
  File "/home/user/bobspear117/bob.spear-1.1.7/bin/evaluate.py", line 21, in <module>
    sys.exit(facereclib.script.evaluate.main())
  File "/usr/lib/python2.7/site-packages/facereclib-1.2.3-py2.7.egg/facereclib/script/evaluate.py", line 168, in main
    scores_dev = [score_parser(os.path.join(args.directory, f)) for f in args.dev_files]
  File "/usr/local/lib64/python2.7/site-packages/bob/measure/load.py", line 58, in split_four_column
    scores_list = four_column(filename)
  File "/usr/local/lib64/python2.7/site-packages/bob/measure/load.py", line 35, in four_column
    (i, filename, l)
SyntaxError: Line 0 of file "./dev/Dcoetzee.hdf5" is invalid: �HDF

the python I am using is:  

Python 2.7.5 (default, Jun 17 2014, 18:11:42)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-16)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
dlopen("/usr/lib64/python2.7/lib-dynload/readline.so", 2);
import readline # dynamically loaded from /usr/lib64/python2.7/lib-dynload/readline.so
>>>

[Manuel Günther]

Dear Martha,

the ```bin/evaluate.py``` script expects score files that have been produced by the speaker verification scripts. These files are clear text files which are usually called "scores-dev" and "scores-eval" (@Elie: correct my if I am wrong).
So, the hdf5 files are definitely wrong. Since you haven't given any details on your experiment (e.g., how you generated the score files), I cannot tell you, where you can find them.

So, could you please provide more details on that?

Thanks
Manuel

[Maria Garcia]

Hi Manuel,
Thank you for your kind reply,

here is the bash script I ran.  It was pretty much out of the box. per the example on the bob.spear page

==================================

$ more simpleOrig.sh

#!/bin/bash

echo "Perform the basic NIST test"

# Source: https://pypi.python.org/pypi/bob.spear

USRDIR=/home/user/bobspear117/bob.spear-1.1.7/OrigUser.111

TMPDIR=/home/user/bobspear117/bob.spear-1.1.7/OrigTmp.111

SCORDT=/home/user/bobspear117/bob.spear-1.1.7/OrigScore.111

./bin/spkverif_gmm.py -d config/database/voxforge.py -p config/preprocessing/energy.py \

 -f config/features/mfcc_60.py -t config/tools/ubm_gmm/ubm_gmm_256G.py -b ubm_gmm -z \

 --user-directory $USRDIR  --temp-directory $TMPDIR 

==================================

I'm not really sure where I should look for scores at.

I find scores here:

$ ls -altr `find OrigTmp.111 -name scores* -print`

total 4

drwxrwxrwx. 7 user user 4096 Feb 20 16:54 ..

drwxrwxrwx. 3 user user   22 Feb 20 16:54 .

drwxrwxrwx. 4 user user   27 Feb 20 16:54 zt_norm_A      * the files in this directory are all HDF5 files    (these are what I looked at first)

$ ls -altr `find OrigUser.111 -name scores-eval -print`

-rw-rw-rw-. 1 user user 203453 Feb 20 16:54 OrigUser.111/ubm_gmm/scores/nonorm/scores-eval

$ ls -altr `find OrigUser.111 -name scores-dev -print`

-rw-rw-rw-. 1 user user 205449 Feb 20 16:54 OrigUser.111/ubm_gmm/scores/nonorm/scores-dev

$ head -n 5 OrigUser.111/ubm_gmm/scores/nonorm/scores-eval

Saul Saul Saul-20080221-arb/wav/b0242 2.83093491877

Saul Saul Saul-20080221-arb/wav/b0243 2.85400989999

Saul Saul Saul-20080221-arb/wav/b0244 2.60969537614

Saul Saul Saul-20080221-arb/wav/b0245 3.43413477493

Saul Saul Saul-20080221-arb/wav/b0246 2.50247076463

$ head -n 5 OrigUser.111/ubm_gmm/scores/nonorm/scores-dev

Dcoetzee Dcoetzee Dcoetzee-20110429-bne/wav/b0530 3.89622904592

Dcoetzee Dcoetzee Dcoetzee-20110429-bne/wav/b0531 5.00382704948

Dcoetzee Dcoetzee Dcoetzee-20110429-bne/wav/b0532 3.42585288762

Dcoetzee Dcoetzee Dcoetzee-20110429-bne/wav/b0533 3.13797232244

Dcoetzee Dcoetzee Dcoetzee-20110429-bne/wav/b0534 5.91024611419

i'm not certain as to whether it crashed now.   I don't know what these fields above mean though.    what is a hit and what is a miss?

============================================================

i tried running with the two files i found above and it crashed but recovered?  the pdf wasn't viewable.  i don't really care about the pdf i think.

$  ~/bobspear117/bob.spear-1.1.7/bin/evaluate.py -e OrigUser.111/ubm_gmm/scores/nonorm/scores-eval -d OrigUser.111/ubm_gmm/scores/nonorm/scores-dev  -c EER -D DET.pdf -x

The EER of the development set of 'OrigUser.111/ubm_gmm/scores/nonorm/scores-dev' is 2.000%

The HTER of the evaluation set of 'OrigUser.111/ubm_gmm/scores/nonorm/scores-dev' is 1.648%

Calibration performance on development set of 'OrigUser.111/ubm_gmm/scores/nonorm/scores-dev' is Cllr 0.76831 and minCllr 0.08580 

Calibration performance on evaluation set of 'OrigUser.111/ubm_gmm/scores/nonorm/scores-dev' is Cllr 0.81069 and minCllr 0.05997

Traceback (most recent call last):

  File "/home/user/bobspear117/bob.spear-1.1.7/bin/evaluate.py", line 21, in <module>

    sys.exit(facereclib.script.evaluate.main())

  File "/usr/lib/python2.7/site-packages/facereclib-1.2.3-py2.7.egg/facereclib/script/evaluate.py", line 246, in main

    pdf.savefig(_plot_det(dets_dev, colors, args.legends if args.legends else args.dev_files, "DET plot for development set"))

  File "/usr/lib64/python2.7/site-packages/matplotlib/backends/backend_pdf.py", line 2258, in savefig

    figure.savefig(self, format='pdf', **kwargs)

  File "/usr/lib64/python2.7/site-packages/matplotlib/figure.py", line 1363, in savefig

    self.canvas.print_figure(*args, **kwargs)

  File "/usr/lib64/python2.7/site-packages/matplotlib/backend_bases.py", line 2093, in print_figure

    **kwargs)

  File "/usr/lib64/python2.7/site-packages/matplotlib/backends/backend_pdf.py", line 2301, in print_pdf

    self.figure.draw(renderer)

  File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper

    draw(artist, renderer, *args, **kwargs)

  File "/usr/lib64/python2.7/site-packages/matplotlib/figure.py", line 999, in draw

    func(*args)

  File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper

    draw(artist, renderer, *args, **kwargs)

  File "/usr/lib64/python2.7/site-packages/matplotlib/axes.py", line 2086, in draw

    a.draw(renderer)

  File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper

    draw(artist, renderer, *args, **kwargs)

  File "/usr/lib64/python2.7/site-packages/matplotlib/legend.py", line 464, in draw

    bbox = self._legend_box.get_window_extent(renderer)

  File "/usr/lib64/python2.7/site-packages/matplotlib/offsetbox.py", line 242, in get_window_extent

    w, h, xd, yd, offsets = self.get_extent_offsets(renderer)

  File "/usr/lib64/python2.7/site-packages/matplotlib/offsetbox.py", line 334, in get_extent_offsets

    whd_list = [c.get_extent(renderer) for c in self.get_visible_children()]

  File "/usr/lib64/python2.7/site-packages/matplotlib/offsetbox.py", line 235, in get_extent

    w, h, xd, yd, offsets = self.get_extent_offsets(renderer)

  File "/usr/lib64/python2.7/site-packages/matplotlib/offsetbox.py", line 392, in get_extent_offsets

    whd_list = [c.get_extent(renderer) for c in self.get_visible_children()]

  File "/usr/lib64/python2.7/site-packages/matplotlib/offsetbox.py", line 235, in get_extent

    w, h, xd, yd, offsets = self.get_extent_offsets(renderer)

  File "/usr/lib64/python2.7/site-packages/matplotlib/offsetbox.py", line 334, in get_extent_offsets

    whd_list = [c.get_extent(renderer) for c in self.get_visible_children()]

  File "/usr/lib64/python2.7/site-packages/matplotlib/offsetbox.py", line 235, in get_extent

    w, h, xd, yd, offsets = self.get_extent_offsets(renderer)

  File "/usr/lib64/python2.7/site-packages/matplotlib/offsetbox.py", line 392, in get_extent_offsets

    whd_list = [c.get_extent(renderer) for c in self.get_visible_children()]

  File "/usr/lib64/python2.7/site-packages/matplotlib/offsetbox.py", line 717, in get_extent

    bbox, info = self._text._get_layout(renderer)

  File "/usr/lib64/python2.7/site-packages/matplotlib/text.py", line 313, in _get_layout

    ismath=ismath)

  File "/usr/lib64/python2.7/site-packages/matplotlib/backends/backend_pdf.py", line 1913, in get_text_width_height_descent

    renderer=self)

  File "/usr/lib64/python2.7/site-packages/matplotlib/texmanager.py", line 603, in get_text_width_height_descent

    dvifile = self.make_dvi(tex, fontsize)

  File "/usr/lib64/python2.7/site-packages/matplotlib/texmanager.py", line 368, in make_dvi

    string:\n%s\nHere is the full report generated by LaTeX: \n\n'% repr(tex)) + report)

RuntimeError: LaTeX was not able to process the following string:

'OrigUser.111/ubm_gmm/scores/nonorm/scores-dev'

Here is the full report generated by LaTeX: 

This is pdfTeX, Version 3.1415926-2.5-1.40.14 (TeX Live 2013)

 restricted \write18 enabled.

entering extended mode

(./5340a4308f4981ee5c72abe31d916280.tex

LaTeX2e <2011/06/27>

Babel <v3.8m> and hyphenation patterns for english, dumylang, nohyphenation, lo

aded.

(/usr/share/texlive/texmf-dist/tex/latex/base/article.cls

Document Class: article 2007/10/19 v1.4h Standard LaTeX document class

(/usr/share/texlive/texmf-dist/tex/latex/base/size10.clo))

(/usr/share/texlive/texmf-dist/tex/latex/type1cm/type1cm.sty)

(/usr/share/texlive/texmf-dist/tex/latex/psnfss/helvet.sty

(/usr/share/texlive/texmf-dist/tex/latex/graphics/keyval.sty))

(/usr/share/texlive/texmf-dist/tex/latex/psnfss/courier.sty)

(/usr/share/texlive/texmf-dist/tex/latex/base/textcomp.sty

(/usr/share/texlive/texmf-dist/tex/latex/base/ts1enc.def))

(/usr/share/texlive/texmf-dist/tex/latex/geometry/geometry.sty

(/usr/share/texlive/texmf-dist/tex/generic/oberdiek/ifpdf.sty)

(/usr/share/texlive/texmf-dist/tex/generic/oberdiek/ifvtex.sty)

(/usr/share/texlive/texmf-dist/tex/generic/ifxetex/ifxetex.sty)

Package geometry Warning: Over-specification in `h'-direction.

    `width' (5058.9pt) is ignored.

Package geometry Warning: Over-specification in `v'-direction.

    `height' (5058.9pt) is ignored.

)

No file 5340a4308f4981ee5c72abe31d916280.aux.

(/usr/share/texlive/texmf-dist/tex/latex/base/ts1cmr.fd)

(/usr/share/texlive/texmf-dist/tex/latex/psnfss/ot1pnc.fd)

*geometry* driver: auto-detecting

*geometry* detected driver: dvips

! Missing $ inserted.

<inserted text> 

                $

l.12 ...00}{27.000000}{\rmfamily OrigUser.111/ubm_

                                                  gmm/scores/nonorm/scores-dev}

! Extra }, or forgotten $.

l.12 ...User.111/ubm_gmm/scores/nonorm/scores-dev}

                                                  

! Missing $ inserted.

<inserted text> 

                $

l.13 \end{document}

                   

[1] (./5340a4308f4981ee5c72abe31d916280.aux) )

(\end occurred inside a group at level 1)

### simple group (level 1) entered at line 12 ({)

### bottom level

(see the transcript file for additional information)

Output written on 5340a4308f4981ee5c72abe31d916280.dvi (1 page, 444 bytes).

Transcript written on 5340a4308f4981ee5c72abe31d916280.log.

[Manuel Günther]

Alright, you have identified the score files correctly. They are always located in the USRDIR.

The current issue you have is related to LaTeX rather than to Python. If not specified otherwise, the directories names are used as labels in the plots. Unfortunately, LaTeX does not like the '_' sign, which it interprets differently, e.g., as math. This is the actual error message that your are receiving.

One simple solution is to provide different labels. AFAIK, the bin/evaluate.py script has an option for that, try "bin/evaluate.py --help" to see the list of options.
Anyways, we should find a solution for these kinds of issues in our code -- at least we should raise an exception with a more clear error message.

@Elie: Do you modify this script in bob.spear, or do you re-use the one from the facereclib?

Best regards
Manuel

[Elie Khoury]

Hello,

Good, looks you solve the problem

@Manuel: I'm calling the script evaluate.py from facereclib.

Elie

--
-- You received this message because you are subscribed to the Google Groups bob-devel group. To post to this group, send email to bob-...@googlegroups.com. To unsubscribe from this group, send email to bob-devel+...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/bob-devel or directly the project website at http://idiap.github.com/bob/
---
You received this message because you are subscribed to the Google Groups "bob-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bob-devel+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Maria Garcia]

Thank you for your help