EER is too high (41.667%)!

[Quazi Marufur Rahman]

Hi,

I am working to run the bob.spear speaker recognition system in my machine using this tutorial.

Please check the attachment for models and probes list, these files are placed in dev folder. I have also attached the Experiments.info and scores-dev file.

Following two commands are being used for verify and evaluation purpose.

bin/verify.py -vvv  -d voxforge -p energy-2gauss -e mfcc-60 -a gmm-voxforge -s ubm_gmm --groups {dev}

 

bin/evaluate.py -d /BobSpear/results/ubm_gmm/nonorm/scores-dev -c EER
The EER of the development set of '/BobSpear/results/ubm-gmm/nonorm/scores-dev' is 41.667%

As you can see, EER is too high (41.667%). Am I missing anything?

What can I do to minimize the EER?

Thanks in advance.

[Tiago Freitas Pereira]

Hi Quazi,

I run the same sequence of commands as you and I got 1.722%

It is quite strange because you are supposed to have 3000 scores in the scores-dev file (as in the attachment) and not only 216.

Furthermore, the scores are supposed to be in the range ~[-5,5].

Your have scores higher than ~600. This makes me think that there is something wrong with your UBM (Universal Background Model).

Could you please run the experiment again from scratch and see if you have the same issue?

Best Regards

--
-- You received this message because you are subscribed to the Google Groups bob-devel group. To post to this group, send email to bob-...@googlegroups.com. To unsubscribe from this group, send email to bob-devel+...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/bob-devel or directly the project website at http://idiap.github.com/bob/
---
You received this message because you are subscribed to the Google Groups "bob-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bob-devel+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Tiago

[Quazi Marufur Rahman]

Hi,

I have run it multiple times and it seems everything worked fine! (./bin/verify.py didn't show any error).

If there is no problem, could you please share your for_models and for_probes file?

Other than this, I think every other configs are same.

One more thing, what is the sampling and bit rate of those .wav files? 

Thanks

You received this message because you are subscribed to a topic in the Google Groups "bob-devel" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bob-devel/rqh-3hYyEnU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bob-devel+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Quazi Marufur Rahman

Department of Computer Science and Engineering

University of Dhaka

Bangladesh

[Tiago Freitas Pereira]

Hi Quazi,

Follow attached the for_models and for_probes from the development set of the database voxforge (https://pypi.python.org/pypi/bob.db.voxforge/2.0.2).

You will see that they are different.

Are you using the bob.bio.spear from here (https://pypi.python.org/pypi/bob.bio.spear)?

The files from Voxforge are sampled in 16KHz.

[Quazi Marufur Rahman]

Hi Tiago,

Many many thanks. It has worked.

However, according to this url, I am trying to run verify.py in parallel mode. However, it's creating an error. Here is the command and corresponding error.

./bin/verify.py -vvv  -d voxforge -p energy-2gauss -e mfcc-60 -a gmm-voxforge -s ubm_gmm --groups 'dev' --parallel 4
verify.py: error: unrecognized arguments: --parallel 4

​Any suggestion?​

 

 

[Manuel Günther]

Indeed, the documentation is wrong. Currently, we have the '--parallel' option only in the './bin/baselines.py' script. This is the one that should be written there, not './bin/verify.py'.
However, you can run in parallel. The command line options are just a bit different. You can use the '--grid' option with a 'local' configuration:
https://pythonhosted.org/bob.bio.base/experiments.html#running-in-parallel
(see also './bin/resources -d grid' for a list of registered 'grid' resources)

So, in your case, the command line options would be: '--grid local-p4 --run-local-scheduler --nice 19 --stop-on-failure', or shorter, '-g local-p4 -LSN 19'.

I am currently working on getting the '--local' option for the './bin/verify.py' script as well, which is just a shortcut for the above command line options.

Cheers
Manuel

--
-- You received this message because you are subscribed to the Google Groups bob-devel group. To post to this group, send email to bob-...@googlegroups.com. To unsubscribe from this group, send email to bob-devel+unsubscribe@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/bob-devel or directly the project website at http://idiap.github.com/bob/

---
You received this message because you are subscribed to the Google Groups "bob-devel" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bob-devel+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Tiago

--
-- You received this message because you are subscribed to the Google Groups bob-devel group. To post to this group, send email to bob-...@googlegroups.com. To unsubscribe from this group, send email to bob-devel+unsubscribe@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/bob-devel or directly the project website at http://idiap.github.com/bob/
---

You received this message because you are subscribed to a topic in the Google Groups "bob-devel" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bob-devel/rqh-3hYyEnU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to bob-devel+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Quazi Marufur Rahman

Department of Computer Science and Engineering

University of Dhaka

Bangladesh

--
-- You received this message because you are subscribed to the Google Groups bob-devel group. To post to this group, send email to bob-...@googlegroups.com. To unsubscribe from this group, send email to bob-devel+unsubscribe@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/bob-devel or directly the project website at http://idiap.github.com/bob/

---
You received this message because you are subscribed to the Google Groups "bob-devel" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bob-devel+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Tiago

--
-- You received this message because you are subscribed to the Google Groups bob-devel group. To post to this group, send email to bob-...@googlegroups.com. To unsubscribe from this group, send email to bob-devel+unsubscribe@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/bob-devel or directly the project website at http://idiap.github.com/bob/

---
You received this message because you are subscribed to a topic in the Google Groups "bob-devel" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bob-devel/rqh-3hYyEnU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to bob-devel+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.