Batch normalization and model collapsing

[Mael Primet]

I did a `CollapseModel` before saving the model and printed the network informations before and after the collapse, the layers before and after the batch normalization layers are identical, the number of batch-normalization layers does not change, but their parameters change (linear-params-rms, bias-{mean,stddev})

is this intended ? I assumed that the layer parameters would change only if two layers are actually collapsed but in this case the number remains the same and the layers before and after are unchanged 

[Mael Primet]

Is the `CollapseModel` method supposed to be idempotent? I'm using it when loading and when saving models, so I'm not sure whether it changes the weights each time twice

[Mael Primet]

After further testing, the behavior seems to come from `SetBatchnormTestMode` because the item count was 0

[Daniel Povey]

OK.
You are only really intended to call CollapseModel() once, after
training the model and before decoding with it, as it does some things
to the model that may not always be ideal if you want to do further
training. But calling it more than once should be harmless.
It tries to get rid of all the batchorm components, as in principle
they are often combinable with the following linearity, but that isn't
possible in all model topologies. There may also be cases where it
*would* be possible for CollapseModel to git rid of the batch-norm
component but it hasn't been extended to do so yet.
In general it should be OK to call SetBatchnormTestMode() twice
without hurting the model, but if you call ZeroComponentStats() and
then SetBatchnormTestMode() without accumulating data, that's a
problem.
Dan

On Mon, Aug 7, 2017 at 7:25 AM, Mael Primet <mael....@snips.ai> wrote:
> After further testing, the behavior seems to come from
> `SetBatchnormTestMode` because the item count was 0
>

> --
> Go to http://kaldi-asr.org/forums.html find out how to join
> ---
> You received this message because you are subscribed to the Google Groups
> "kaldi-help" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kaldi-help+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.