how is batch normalization layer working in multi-gpu mode?

[Etienne Perot]

Hello,

I would like to know how BN layer accumulates mean & variance statistics in multi-gpu mode, and if it is possible to use the same principle when batches are too small for low-budget scenarii like Fully-Convolutional Mode? I am assuming Gradient Accumulation wouldn't do the job, since it is not analytically equivalent.

  

[Prasanna S]

I don't think mean and variance are learnable parameters and thus update might not be called on them. I am also wondering about the same thing. If you find something, do let u