I have two questions about UBM training in SPEAR.
The first one is training UBM took extremely long time. When I used 600 hours speech data to train a UBM (256g) with SPEAR, it took me about five days. Is it normal to every one? Or I may have some mistakes in using it.
The second question is whether I can use UBM trained by HTK in SPEAR. I think using HTK to train the UBM with the same data will take less time than 5 days. I know in feature extraction step, SPEAR has the function of HTK Feature reader. So I would like to ask whether there is similar function in UBM training step such as HTK UBM reader?
Looking forward your reply! Thank you!
Regards,
Qiongqiong
I actually have tried to use the parallel implementation of GMM "$ bin/para_ubm_spkverif_gmm.py", but i met some problems. I would appreciate it if you could help me on this issue also.
I have tried only to replace “spkverif_gmm.py” with “para_ubm_spkverif_gmm.py” and kept the options same as the following line. It could work but the processing time was almost same as when I just use “spkverif_gmm.py”. So I think there must be something wrong so that it actually didn’t do the process parallelly.
$ ./bin/ para_ubm_spkverif_gmm.py \
-d config/database/voxforge.py \
-p config/preprocessing/energy.py \
-f config/features/mfcc_60.py \
-t config/tools/ubm_gmm/ubm_gmm_256G.py \
-b ubm_gmm -z \
--user-directory PATH/TO/USER/DIR--temp-directory PATH/TO/TEMP/DIR
By comparing the usages of the two commands - spkverif_gmm.py and $ bin/para_ubm_spkverif_gmm.py, I found the following different options which are only in “para_ubm_spkverif_gmm.py”:
[-l LIMIT_TRAINING_EXAMPLES]
[-K KMEANS_TRAINING_ITERATIONS]
[-k KMEANS_START_ITERATION]
[-M GMM_TRAINING_ITERATIONS]
[-m GMM_START_ITERATION] [-n] [-C]
[--skip-normalization] [--skip-k-means]
[--skip-gmm]
I think they might be necessary for the parallel training. But I am not clear how to use them. Could you give some brief explanation and a examples? Sorry for taking your time! Thank you!
Best Regards,
Qiongqiong
$ ./bin/para_ubm_spkverif_gmm.py -d config/database/voxforge.py -p config/preprocessing/energy.py \ -f config/features/mfcc_60.py -t config/tools/ubm_gmm/ubm_gmm_256G.py -b ubm_gmm -z \ --user-directory PATH/TO/USER/DIR --temp-directory PATH/TO/TEMP/DIR -g config/grid/para_training_local.py
$ bin/jman --local -vv run-scheduler --parallel 6In this example, the number of nodes is 6
$ ./bin/para_ubm_spkverif_gmm.py -d config/database/voxforge.py -p config/preprocessing/energy.py \ -f config/features/mfcc_60.py -t config/tools/ubm_gmm/ubm_gmm_256G.py -b ubm_gmm -z \ --user-directory PATH/TO/USER/DIR --temp-directory PATH/TO/TEMP/DIR -g config/grid/para_training_sge.py
--
-- You received this message because you are subscribed to the Google Groups bob-devel group. To post to this group, send email to bob-...@googlegroups.com. To unsubscribe from this group, send email to bob-devel+...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/bob-devel or directly the project website at http://idiap.github.com/bob/
---
You received this message because you are subscribed to the Google Groups "bob-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bob-devel+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
-- ------------------- Dr. Elie Khoury Post Doctorant Biometric Person Recognition Group IDIAP Research Institute (Switzerland) Tel : +41 27 721 77 23
Dear Manuel,
yes, I use bob.bio.spear. I tried the --parallel N
option, and got the error bellow.
I read somewhere that this may be because the program tries to open for writing a dictionary where I don’t have the permission but I don’t know which directory this may be as I have write access for the “ sub-directory”.
Could you tell me, what the problem is?
What is tconfig/grid/para_training_local.py
? Where can I find it?
Thanks
Márton
(venv)makrai@ron:/mnt/permanent/Language$ time python -m bob.bio.base.script.verify -d ~/repo/hunspeech/emLid/babel.py -p energy-2gauss -e mfcc-60 -a gmm -vvs /mnt/store/makrai/work/emLid/spear_babel/ --grid local-p4 --run-local-scheduler --nice 10
bob.bio.base@2016-08-26 10:10:34,358 -- ERROR: During the execution, an exception was raised: (sqlite3.OperationalError) unable to open database file
Traceback (most recent call last):
File "/home/makrai/tool/python/venv/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect
return fn()
File "/home/makrai/tool/python/venv/lib/python3.4/site-packages/sqlalchemy/pool.py", line 376, in connect
return _ConnectionFairy._checkout(self)
File "/home/makrai/tool/python/venv/lib/python3.4/site-packages/sqlalchemy/pool.py", line 713, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/home/makrai/tool/python/venv/lib/python3.4/site-packages/sqlalchemy/pool.py", line 480, in checkout
rec = pool._do_get()
File "/home/makrai/tool/python/venv/lib/python3.4/site-packages/sqlalchemy/pool.py", line 1151, in _do_get
return self._create_connection()
File "/home/makrai/tool/python/venv/lib/python3.4/site-packages/sqlalchemy/pool.py", line 323, in _create_connection
return _ConnectionRecord(self)
File "/home/makrai/tool/python/venv/lib/python3.4/site-packages/sqlalchemy/pool.py", line 449, in __init__
self.connection = self.__connect()
File "/home/makrai/tool/python/venv/lib/python3.4/site-packages/sqlalchemy/pool.py", line 607, in __connect
connection = self.__pool._invoke_creator(self)
File "/home/makrai/tool/python/venv/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py", line 97, in connect
return dialect.connect(*cargs, **cparams)
File "/home/makrai/tool/python/venv/lib/python3.4/site-packages/sqlalchemy/engine/default.py", line 385, in connect
return self.dbapi.connect(*cargs, **cparams)
sqlite3.OperationalError: unable to open database file
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3.4/runpy.py", line 170, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.4/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/makrai/tool/python/venv/lib/python3.4/site-packages/bob/bio/base/script/verify.py", line 453, in <module>
main()
File "/home/makrai/tool/python/venv/lib/python3.4/site-packages/bob/bio/base/script/verify.py", line 446, in main
verify(args, command_line_parameters)
File "/home/makrai/tool/python/venv/lib/python3.4/site-packages/bob/bio/base/script/verify.py", line 399, in verify
retval = add_jobs(args, submitter)
File "/home/makrai/tool/python/venv/lib/python3.4/site-packages/bob/bio/base/script/verify.py", line 57, in add_jobs
**args.grid.preprocessing_queue)
File "/home/makrai/tool/python/venv/lib/python3.4/site-packages/bob/bio/base/tools/grid.py", line 111, in submit
**kwargs
File "/home/makrai/tool/python/venv/lib/python3.4/site-packages/gridtk/local.py", line 48, in submit
self.lock()
File "/home/makrai/tool/python/venv/lib/python3.4/site-packages/gridtk/manager.py", line 61, in lock
self._create()
File "/home/makrai/tool/python/venv/lib/python3.4/site-packages/gridtk/manager.py", line 86, in _create
Base.metadata.create_all(self._engine)
File "/home/makrai/tool/python/venv/lib/python3.4/site-packages/sqlalchemy/sql/schema.py", line 3695, in create_all
tables=tables)
File "/home/makrai/tool/python/venv/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 1855, in _run_visitor
with self._optional_conn_ctx_manager(connection) as conn:
File "/usr/lib/python3.4/contextlib.py", line 59, in __enter__
return next(self.gen)
File "/home/makrai/tool/python/venv/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 1848, in _optional_conn_ctx_manager
with self.contextual_connect() as conn:
File "/home/makrai/tool/python/venv/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 2039, in contextual_connect
self._wrap_pool_connect(self.pool.connect, None),
File "/home/makrai/tool/python/venv/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect
e, dialect, self)
File "/home/makrai/tool/python/venv/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 1405, in _handle_dbapi_exception_noconnection
exc_info
File "/home/makrai/tool/python/venv/lib/python3.4/site-packages/sqlalchemy/util/compat.py", line 202, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/home/makrai/tool/python/venv/lib/python3.4/site-packages/sqlalchemy/util/compat.py", line 185, in reraise
raise value.with_traceback(tb)
File "/home/makrai/tool/python/venv/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect
return fn()
File "/home/makrai/tool/python/venv/lib/python3.4/site-packages/sqlalchemy/pool.py", line 376, in connect
return _ConnectionFairy._checkout(self)
File "/home/makrai/tool/python/venv/lib/python3.4/site-packages/sqlalchemy/pool.py", line 713, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/home/makrai/tool/python/venv/lib/python3.4/site-packages/sqlalchemy/pool.py", line 480, in checkout
rec = pool._do_get()
File "/home/makrai/tool/python/venv/lib/python3.4/site-packages/sqlalchemy/pool.py", line 1151, in _do_get
return self._create_connection()
File "/home/makrai/tool/python/venv/lib/python3.4/site-packages/sqlalchemy/pool.py", line 323, in _create_connection
return _ConnectionRecord(self)
File "/home/makrai/tool/python/venv/lib/python3.4/site-packages/sqlalchemy/pool.py", line 449, in __init__
self.connection = self.__connect()
File "/home/makrai/tool/python/venv/lib/python3.4/site-packages/sqlalchemy/pool.py", line 607, in __connect
connection = self.__pool._invoke_creator(self)
connection = self.__pool._invoke_creator(self)
File "/home/makrai/tool/python/venv/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py", line 97, in connect
return dialect.connect(*cargs, **cparams)
File "/home/makrai/tool/python/venv/lib/python3.4/site-packages/sqlalchemy/engine/default.py", line 385, in connect
return self.dbapi.connect(*cargs, **cparams)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) unable to open database file
--
-- You received this message because you are subscribed to the Google Groups bob-devel group. To post to this group, send email to bob-...@googlegroups.com. To unsubscribe from this group, send email to bob-devel+unsubscribe@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/bob-devel or directly the project website at http://idiap.github.com/bob/
---
You received this message because you are subscribed to the Google Groups "bob-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bob-devel+unsubscribe@googlegroups.com.
--
-- You received this message because you are subscribed to the Google Groups bob-devel group. To post to this group, send email to bob-...@googlegroups.com. To unsubscribe from this group, send email to bob-devel+...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/bob-devel or directly the project website at http://idiap.github.com/bob/
---
You received this message because you are subscribed to the Google Groups "bob-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bob-devel+...@googlegroups.com.
--Tiago
sorry, I repeat the error formatted
On Friday, August 26, 2016 at 12:34:22 PM UTC+2, Márton Makrai wrote:
Thank you very much, in the help I learned that the default for the gridtk-database-file is in the working directory (where the python command is run), so this is solved.
Unfortunately now I get an other error.
I tried
`python -m bob.bio.base.script.verify -d ~/repo/hunspeech/emLid/babel.py -p energy-2gauss -e mfcc-60 -a gmm -vvs /mnt/store/makrai/work/emLid/spear_babel/ --nice 10 --parallel 4`
and
`python -m bob.bio.base.script.verify -d ~/repo/hunspeech/emLid/babel.py -p energy-2gauss -e mfcc-60 -a gmm -vvs /mnt/store/makrai/work/emLid/spear_babel/ --grid local-p4 --run-local-scheduler --nice 10 --parallel 4`
and I got similar error.
./bin/verify.py -d ~/repo/hunspeech/emLid/babel.py -p energy-2gauss -e mfcc-60 -a gmm -vvs /mnt/store/makrai/work/emLid/spear_babel/ --nice 10 --parallel 4