Training ivector extractor on development dataset in sre08 taking very long.

[abhishek...@quantiphi.com]

Hi,

Could anyone explain why ivector extractor training on the development set in sre08 is taking so long, its been more than 15 hours, on a 32 CPU 208 GB Ram machine. 

I trained the UBM on whole development set and not a subset. Is this a valid move or does it require only a subset of data ?

[Daniel Povey]

Using the whole development set to train the UBM is fine.
i-vector extractor training is expected to take a while.
It may be trying to use more CPUs than you have available (if you used
the default num-jobs, num-threads, etc., and you are using run.pl).
You could decrease some of those things but it may not improve the
speed much (depends whether memory is thrashing).

Dan

> This message contains information that may be privileged or confidential and
> is the property of the Quantiphi Inc and/or its affiliates. It is intended
> only for the person to whom it is addressed. If you are not the intended
> recipient, any review, dissemination, distribution, copying, storage or
> other use of all or any portion of this message is strictly prohibited. If
> you received this message in error, please immediately notify the sender by
> reply e-mail and delete this message in its entirety.
>
> --
> Go to http://kaldi-asr.org/forums.html find out how to join
> ---
> You received this message because you are subscribed to the Google Groups
> "kaldi-help" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kaldi-help+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[abhishek...@quantiphi.com]

Thank you, I saw the train_ivector_extractor.sh script, and ran parallel jobs accordingly.

[Santiago Prieto]

Hi everyone,

I have the same problem training a ivector extractor model because my computer has only 4 CPU's. But it has GPU's and I'm wondering if it is possible to train with a GPU. I've been looking in kaldi/src/ files and I didn't find any code to be able to train with GPU, isn't it? If someone can give some tips, I would like to program it for GPU's training.

Thanks in advance.

[Abhishek Dandona]

I believe the way kaldi is designed is to use least amount of resources with max utilization. I have only come across neural network code GPU compatible, everything else uses CPU.

--

Go to http://kaldi-asr.org/forums.html find out how to join
---

You received this message because you are subscribed to a topic in the Google Groups "kaldi-help" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kaldi-help/k4bTdnP0rzE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kaldi-help+unsubscribe@googlegroups.com.
To post to this group, send email to kaldi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/c1efefdd-6d05-420c-b4d7-ba7c1552ce94%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Daniel Povey]

No it's not possible to train the ivector extractor with a GPU.

I think even if we had implemented that, there might be problems with memory consumption; IIRC the stats are larger than the 8G max of memory that a GPU will normally have.

Dan

You received this message because you are subscribed to the Google Groups "kaldi-help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+unsubscribe@googlegroups.com.

To post to this group, send email to kaldi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/CA%2BSOy8g_A9VSJ167XaTvD1VKVy8sPpZUWXKYnP%2BSfX4qhoQQ3w%40mail.gmail.com.