GMMTrainer logadd exception

[Benjamin Helgert]

Hey guys,

I just got this error message again while training UBMs. 3 in 4 UBMs were already trained without any problems with the same exact code but now I get the following exception:  

Traceback (most recent call last):

  File "/home/bhelgert/PycharmProjects/adding_and_adjusting_labels/scratch.py", line 237, in <module>

    train_ubms( data_dir, max_file_count = 2000, mixture_count = 64 )

  File "/home/bhelgert/PycharmProjects/adding_and_adjusting_labels/scratch.py", line 210, in train_ubms

    bob.learn.em.train( trainer, gmm, mfcc_features, max_iterations = 200, convergence_threshold = 1e-5 )

  File "/home/bhelgert/miniconda3/envs/bob_env_py27/lib/python2.7/site-packages/bob/learn/em/train.py", line 51, in train

    trainer.e_step(machine, data)

RuntimeError: bob.learn.em.ML_GMMTrainer - cannot perform the e_step method: C++ exception caught: 'logadd: minusdif (-nan) log_b (-nan) or log_a (-56.477783) is nan'

Process finished with exit code 1

I had this problem before and solved it by using more training data but now it appeared again. Does anyone know what can cause this and what is a clean fix for it?

Any help is appreciated.

Best

Benjamin

[Tiago Freitas Pereira]

Hi Benjamin,

Try to search for some abnormalities (nan, inf, etc..) in the GMM means, variances and weights at every iteration (before the e_step).

Check your inputs too.

Basically you are having nan in the log likelihood computation.

Cheers

--
-- You received this message because you are subscribed to the Google Groups bob-devel group. To post to this group, send email to bob-...@googlegroups.com. To unsubscribe from this group, send email to bob-devel+unsubscribe@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/bob-devel or directly the project website at http://idiap.github.com/bob/
---
You received this message because you are subscribed to the Google Groups "bob-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bob-devel+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Tiago

[Benjamin Helgert]

Hey,

Thanks for your quick reply!

Well as of now I only call GMMTrainer once and let it do its thing. I know there is a way to call the E and M steps separately and iteratively. Is there any documentation about how to do this? I did not find anything about this yet and was looking into the bob code to try and figure out how this is done.

Best
Benjamin

[Tiago Freitas Pereira]

Hi Benjamin,

Instead of using the function `bob.learn.em.train`, you can do something like this:

for i in range(ITERATIONS):
    your_trainer.initialize(gmm_machine, data)
    your_trainer.e_step(gmm_machine, data)

    print(gmm_machine.means)

    your_trainer.m_step(gmm_machine, data)

Best
Benjamin

--
-- You received this message because you are subscribed to the Google Groups bob-devel group. To post to this group, send email to bob-...@googlegroups.com. To unsubscribe from this group, send email to bob-devel+unsubscribe@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/bob-devel or directly the project website at http://idiap.github.com/bob/
---
You received this message because you are subscribed to the Google Groups "bob-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bob-devel+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Tiago