Why header size is at most 512k?

[franc...@clustertech.com]

Hi,

I am using cwebp to do image compression. I find that m0, m1, m2 and m3 will return error about the partition #0 is too big to fit 512k. However, m4 does not terminate for three days.

I know that https://groups.google.com/a/webmproject.org/forum/#!topic/webp-discuss/utRtxFLYmPk may solve the problem. But i found that the mode selection will use I16. Thus, degenerating to m1. Refer to the changes, the most important step is adding "mb_header_limit_" to inference the mode selection procedure.

I would like to know that how mb_header_limit_ could help to restrict the header size to 512k?

I have tried to modified the code any using deduct the I4 mode cost from mb_header_limit_ (256 * 510 * 8 * 1024). When it is positive, it choose I16 and I4 by residuals. When it is near zero, only I16 will be chosen. However, I find that the total cost is less than mb_header_limit_ but there is still an error about the partition #0 is too big to fit 512k.

Moreover, why the header need to fit 512k? Is it an any industrial standard?

Thanks,

Francis

[Pascal Massimino]

Hi Francis,

On Wed, Sep 14, 2016 at 9:04 AM, <franc...@clustertech.com> wrote:

Hi,

I am using cwebp to do image compression. I find that m0, m1, m2 and m3 will return error about the partition #0 is too big to fit 512k. However, m4 does not terminate for three days.

oh? This is likely a bug. Could you send (privately?) the offending input picture that is problematic? That would help debugging. We shouldn't loop indefinitely.

 

I know that https://groups.google.com/a/webmproject.org/forum/#!topic/webp-discuss/utRtxFLYmPk may solve the problem. But i found that the mode selection will use I16. Thus, degenerating to m1. Refer to the changes, the most important step is adding "mb_header_limit_" to inference the mode selection procedure.

I would like to know that how mb_header_limit_ could help to restrict the header size to 512k?

I have tried to modified the code any using deduct the I4 mode cost from mb_header_limit_ (256 * 510 * 8 * 1024). When it is positive, it choose I16 and I4 by residuals. When it is near zero, only I16 will be chosen. However, I find that the total cost is less than mb_header_limit_ but there is still an error about the partition #0 is too big to fit 512k.

Yes,. mb_header_limit_ was deliberately set to be coarse, to allow for variation (some macroblock use way more bits that others. So a tight limit would be overly detrimental).

Moreover, why the header need to fit 512k? Is it an any industrial standard?

This comes from the VP8 standard, which itself imposed this limit for hardware reasons. Partition0 needs to stay resident during the decoding  process, hence there must be some limit.

hope it helps,

skal/

[franc...@clustertech.com]

Hi skal,

I am pleased with your quick response.

I could give you two samples, call big1.jpg and big2.jpg. (send in another channel because of size of files)

Both two pictures fail in m2 with the partition #0 is too big to fit 512k and do not terminate in m4 for days.

When I do some hacking on mode selection which following :

 - using only DC mode for UV selection

 - using only I16 with DC mode for Y selection

I found that big1.jpg could produce without error while big2.jpg could not.

I think that big2.jpg may be a case that no webp could be produced currently because the smallest mode head has been chosen (even setting quality = 0 and partition_limit = 100). 

Thanks,

Francis Fok

[Pascal Massimino]

Hi Francis,

On Wed, Sep 14, 2016 at 7:17 PM, <franc...@clustertech.com> wrote:

Hi skal,

I am pleased with your quick response.

I could give you two samples, call big1.jpg and big2.jpg. (send in another channel because of size of files)

Both two pictures fail in m2 with the partition #0 is too big to fit 512k and do not terminate in m4 for days 

Thanks for sending the file! big2.jpg effectively triggered a bug, which should be fixed by this patch:

  https://chromium-review.googlesource.com/#/c/385378/

This will at least prevent the infinite loop.

When I do some hacking on mode selection which following :

 - using only DC mode for UV selection

 - using only I16 with DC mode for Y selection

I found that big1.jpg could produce without error while big2.jpg could not.

I think that big2.jpg may be a case that no webp could be produced currently because the smallest mode head has been chosen (even setting quality = 0 and partition_limit = 100). 

big2.jpg is quite a challenging picture, and we might need to introduce an 'emergency_coding' mode when all other

options have be exhausted for trying to reduce the header size. This emergency coding would include using only

DC mode, I16 DC, and maybe more, even if quality will suffer.

thanks again for finding this bug!

skal/

 

Thanks,

Francis Fok

On Thursday, 15 September 2016 02:44:20 UTC+8, skal wrote:

Hi Francis,

On Wed, Sep 14, 2016 at 9:04 AM, <franc...@clustertech.com> wrote:

Hi,

I am using cwebp to do image compression. I find that m0, m1, m2 and m3 will return error about the partition #0 is too big to fit 512k. However, m4 does not terminate for three days.

oh? This is likely a bug. Could you send (privately?) the offending input picture that is problematic? That would help debugging. We shouldn't loop indefinitely.

 

I know that https://groups.google.com/a/webmproject.org/forum/#!topic/webp-discuss/utRtxFLYmPk may solve the problem. But i found that the mode selection will use I16. Thus, degenerating to m1. Refer to the changes, the most important step is adding "mb_header_limit_" to inference the mode selection procedure.

I would like to know that how mb_header_limit_ could help to restrict the header size to 512k?

I have tried to modified the code any using deduct the I4 mode cost from mb_header_limit_ (256 * 510 * 8 * 1024). When it is positive, it choose I16 and I4 by residuals. When it is near zero, only I16 will be chosen. However, I find that the total cost is less than mb_header_limit_ but there is still an error about the partition #0 is too big to fit 512k.

Yes,. mb_header_limit_ was deliberately set to be coarse, to allow for variation (some macroblock use way more bits that others. So a tight limit would be overly detrimental).

Moreover, why the header need to fit 512k? Is it an any industrial standard?

This comes from the VP8 standard, which itself imposed this limit for hardware reasons. Partition0 needs to stay resident during the decoding  process, hence there must be some limit.

hope it helps,

skal/

--

You received this message because you are subscribed to the Google Groups "WebP Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to webp-discuss+unsubscribe@webmproject.org.
To post to this group, send email to webp-d...@webmproject.org.
Visit this group at https://groups.google.com/a/webmproject.org/group/webp-discuss/.
For more options, visit https://groups.google.com/a/webmproject.org/d/optout.