Question about the necessity of hyperprior

[Isaac Gerg]

Hi,

I see in "Variational image compression with a scale hyperprior" that
the hyperprior is used to take the out spatially varying entropy
(variance) of the sample image in Figure 2 left middle. Why not just
normalize all the variances of the features at each spatial location?
My thought would be that that should eliminate the hyperprior and the
encoding optimization can be handled by the network itself. I would
assume though that have already tried this or decided against it from
reading your papers (they are very good). So, what am I missing? :)

Thank you,
Isaac

[Isaac Gerg]

I'm going to take a stab and answer my own question.   I suppose what I propose is easily shown to be suboptimal.  For example, you may have one section of an image which predicts with almost certainty that the rest of it might be blank.  In this case, you don't want to encode the blank area using the same pdf as other areas.  This means you must have quite a few of these tables for the hyperprior if you model every latent variable (16x16x128 for example if your input image is 256x256).  Is this correct?

Isaac

[Johannes Ballé]

Hi Isaac,

the long answer depends on what you mean exactly by "normalizing the variances of the features". For the purposes of a short answer, I'm going to assume you mean to rescale the features such that their variance is one everywhere. The problem with that is that if you normalize the variance, but keep the quantization step size the same, then you will encode low-variance regions with much higher precision than necessary. In addition, if the encoder normalizes all the features to unit variance, how would the decoder undo the normalization without receiving additional information from the encoder? It would not have access to the original variances, unless the encoder transmits it.

The hyperprior architecture represents exactly that: information about the variances that the encoder shares with the decoder.

Johannes.

--
You received this message because you are subscribed to the Google Groups "tensorflow-compression" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tensorflow-compre...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tensorflow-compression/423be841-c7cc-4317-b15c-5499b45d2ddfn%40googlegroups.com.

[Isaac Gerg]

Thanks Johanne, this was very helpful.

Another question, when you estimate the parameters of the hyperprior,
how is their error backpropagated? For example, if you assign a
certain set of distribution parameters to a particular latent
variable, how do you know the estimate is correct? It looks like
your hyperprior params are used to seed the table of the AE/AD. But,
none of the outputs of those modules (AE/AD) are used during training
time. You pass through the latent code through by adding uniform
noise and you use the hyperprior params along with the pdf of the
latent code to estimate the entropy/rate. What prevents the hyperprior
network setting all the variances of the latents to zero and thereby
reducing the rate. I am uncertain how the parameters of the
hyperprior network affect network training other than what I just
mentioned. In this case, I don't see why the network wouldn't "cheat"
like I mentioned. I must be missing something obvious.

Isaac

On Wed, Feb 3, 2021 at 9:58 PM 'Johannes Ballé' via
tensorflow-compression <tensorflow-...@googlegroups.com>
wrote:

> To view this discussion on the web visit https://groups.google.com/d/msgid/tensorflow-compression/CAN4xYSBXHJpsC5BNYJzHnt5P%2B-2043Mzoz4aD_1L41of_v8q1w%40mail.gmail.com.

[Isaac Gerg]

Hi Johanne,

Does anything prevent the hyperpriors from being all mu=0, sigma=1 and letting the latents simply take on their own mu and sigma bypassing the hyperprior model?

Isaac

[Johannes Ballé]

Hi Isaac,

the model will not cheat because it still needs to balance the rate term(s) with the distortion term. Like in the 2017 model, the rate term fits the entropy model in a maximum likelihood sense to the latent space activations. The latents in turn cannot become all zero due to the distortion term (unless the rate term has a much higher weight).

Hope this helps!

Johannes

To view this discussion on the web visit https://groups.google.com/d/msgid/tensorflow-compression/a9b04814-576b-4841-9d32-293d4bdc1e5fn%40googlegroups.com.

[Isaac Gerg]

Thanks Johannes,  It all clicks now and I was able to get everything working from scratch and I believe I understand all the parts of your code now.  This is a very interesting problem and I think your approach is very interesting and  fun to study. -Isaac

[Johannes Ballé]

Glad to hear!

To view this discussion on the web visit https://groups.google.com/d/msgid/tensorflow-compression/9547bcb5-b5e4-4e2c-90d3-e8c97dce0dacn%40googlegroups.com.