optimizing model load time?

[David van Leeuwen]

Hello, 

I've noticed that the time loading a nnet3 chain acoustic model is a lot larger than for the old nnet2 models.  I realize in the nnet3 setup the computation graph needs to be compiled, but the times that Kaldi reports in spending that are only a fraction of the total time spent completing loading the model.   A small table:

model -- loop compile -- total model load time -- decoding RTF

nnet2-ms -- NA -- 1.1 s -- 0.239

nnet3-tdnn -- 0.25 s -- 8.9 s -- 0.106

nnet3-tdnn-lstm -- 1.32 -- 13.9 s -- 0.249

Al models have been trained using the same data, nnet3 models are somewhat more complex than the nnet2 model. 

Obviously for ASR development the model loading time is insignificant w.r.t. the decoding time, but for our deployment 13 sec start up time is going to become a problem for the user experience.  

I don't understand where the discrepancy between the kaldi-reported compilation time and the actual loading time.  With statement-by-statement timing it seems the call `decodable_info3 = new kaldi::nnet3::DecodableNnetSimpleLoopedInfo(*decodable_opts3, &am_nnet3);` takes up all the time, where approx 10% of that is reported as loop compilation.  

Is there a way we can speed up the loading process by storing a pre-compiled network?

Cheers, 

---david

[Daniel Povey]

Can you please test whether the patch here

https://github.com/kaldi-asr/kaldi/pull/2180

fixes your problem, and leaves the results unchanged?  It should.

It turns out that ComputeSimpleNnetContext() was taking time proportional to the chunk size,

for complicated reasons; and as it happens you were setting the frames-per-chunk to

an unusually large value (which I mentioned in another conversation).  That's probably why

the slowness was noticeable for you.

Dumping a model which you've copied with nnet3-am-copy with the --prepare-for-test=true option might

also slightly improve the speed of reading the model.  I added that option to that binary in the same 

pull request (it was previously only in nnet3-copy).

Dan

--
Go to http://kaldi-asr.org/forums.html find out how to join
---
You received this message because you are subscribed to the Google Groups "kaldi-help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+unsubscribe@googlegroups.com.
To post to this group, send email to kaldi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/d362c3b1-24af-41c5-8ead-7a9cea212681%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Daniel Povey]

... and regarding what you were saying about deployment: you'd normally want to write some kind of server code that starts up and waits for requests to process... having the decoder program start up for each incoming request wouldn't make sense.

[David van Leeuwen]

Hi, 

On Wednesday, January 24, 2018 at 12:40:15 AM UTC+1, Dan Povey wrote:

... and regarding what you were saying about deployment: you'd normally want to write some kind of server code that starts up and waits for requests to process... having the decoder program start up for each incoming request wouldn't make sense.

We start up a session process on the server as soon as a client starts, but the client conditions determine the model (e.g., language), so only then we can load the model.  Then the session stays alive until the client has stopped sending recognition requests for a while.  

If you would not have fixed this just now, we might have to change the server setup to work with a pool of recognition servers for the various languages and conditions, which would definitely complicate our backend design.  

Thanks, 

---david

 

To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.

[Anjos]

Dear Dan, Hello, Can I ask you a problem?

I have a effort on optimizing model load time， and now this chain model load time is 10 senconds.

I also use nnet3-am-copy --prepare-for-test=true to let model size slight, but the model load time is also 10 senconds.

my parameters is as follows:

--config=exp/chain/tdnn_1a_online/conf/online.conf --frames-per-chunk=150 --extra-left-context-initial=0 --online=true --frame-subsampling-factor=3 --max-active=7000 --min-active=200 --beam=10  --acoustic-scale=1.0 --chunk-length=0.5

I see this frames-per-chunk affect the model loading time.set this value lower can speed up the model load time? and it does not affect the model results?

or, can I ask you have another method to optimize model load time？

[Daniel Povey]

I don't know (e.g. what program you are trying to load it to).. I had never noticed it being slow.  Could be disk I/O?  (Dunno.)

frames-per-chunk does not affect that.

--

Go to http://kaldi-asr.org/forums.html find out how to join
---
You received this message because you are subscribed to the Google Groups "kaldi-help" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/f0c63fc0-37a7-4b92-bfc8-484c27fc962eo%40googlegroups.com.

[Anjos]

2020-07-17 17:44:29.616

2020-07-17 17:44:29.616 init opt 0 ms

2020-07-17 17:44:29.619 load fetrue pipe 3 ms

2020-07-17 17:44:29.619 read hmm in the final.mdl 0 ms

2020-07-17 17:44:34.778 read nnet3 in the final.mdl  5159 ms

2020-07-17 17:44:34.782 set nnet3 3 ms

LOG (kaldi-wake-word.exe[5.5-win]:kaldi::nnet3::Nnet::RemoveOrphanNodes():nnet3\nnet-nnet.cc:948) Removed 1 orphan nodes.

LOG (kaldi-wake-word.exe[5.5-win]:kaldi::nnet3::Nnet::RemoveOrphanComponents():nnet3\nnet-nnet.cc:847) Removing 2 orphan components.

LOG (kaldi-wake-word.exe[5.5-win]:kaldi::nnet3::ModelCollapser::Collapse():nnet3\nnet-utils.cc:1472) Added 1 components, removed 2

2020-07-17 17:44:34.816 collapse nnet3 34 ms

2020-07-17 17:44:34.816 config net 0 ms

LOG (kaldi-wake-word.exe[5.5-win]:kaldi::nnet3::CompileLooped():nnet3\nnet-compile-looped.cc:345) Spent 4.94883 seconds in looped compilation.

2020-07-17 17:44:44.627 load DecodableNnetSimpleLoopedInfo 9812 ms

2020-07-17 17:44:44.628 load HCLG 0 ms

2020-07-17 17:44:44.628 load words 0 ms

2020-07-17 17:44:44.628 init Ivector extractor 1 ms

2020-07-17 17:44:44.628 15012 ms

2020-07-17 17:44:34.778 read nnet3 in the final.mdl  5159 ms is as follow:

```cpp

am_nnet->Read(ki.Stream(), binary);

```

2020-07-17 17:44:44.627 load DecodableNnetSimpleLoopedInfo 9812 ms

```cpp

nnet3::DecodableNnetSimpleLoopedInfo *decodable_info = new nnet3::DecodableNnetSimpleLoopedInfo(*decodable_opts, am_nnet);

```

I find that change frames-per-chunk value can improve the load time. and then, the above is the load model time.

I want to continue to improve the load time.

Can I ask, is there any other method to improve?

[Daniel Povey]

Ah, it spends some time in the compilation,

There are ways to dump that computation to disk and read it back, the training programs have 'cache'-related options that you could adapt for this use-case.  Or just try to dump the NnetComputation object and read it back without using an explicit notion of a cache.

--
Go to http://kaldi-asr.org/forums.html find out how to join
---
You received this message because you are subscribed to the Google Groups "kaldi-help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/607c6a2b-df77-4a0b-b1e3-ad278ebbde2do%40googlegroups.com.

[Anjos]

o, my gad, thank you for Dan and quick reply, and, I will try to dump the NnetComputation object. and.

in the 'cache'-related option, I find that it is in the ./nnet3/nnet-chain-training.cc. it it so interesting!!!

[Immu Ali]

Hi Anjos ,

It'll be helpful providing the process of NnetComputation object.

Thanks in advance.

On Sat, 18 Jul, 2020, 1:42 PM Anjos, <aqh...@gmail.com> wrote:

o, my gad, thank you for Dan and quick reply, and, I will try to dump the NnetComputation object. and.

in the 'cache'-related option, I find that it is in the ./nnet3/nnet-chain-training.cc. it it so interesting!!!

--

Go to http://kaldi-asr.org/forums.html find out how to join
---
You received this message because you are subscribed to the Google Groups "kaldi-help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/12486b3a-ad7e-4296-ac73-884d55e32413o%40googlegroups.com.