PNG to AVIF results in larger file size w/ slower speed

[Olzhas Alexandrov]

To reproduce the, I guess, bug, please use:
- `libpng-devel` (ver. 1.2.49)
- `libheif` (ver. 1.13.0)
- `libaom` (ver. 3.5.0)

- `example.png` (attached zip to avoid pre-processing on this website)

Run this command to convert PNG to AVIF:
```sh
heif-enc --avif -p quality=64 -p speed=0 -o 0.avif example.png &&
heif-enc --avif -p quality=64 -p speed=1 -o 1.avif example.png &&
heif-enc --avif -p quality=64 -p speed=2 -o 2.avif example.png &&
heif-enc --avif -p quality=64 -p speed=3 -o 3.avif example.png &&
heif-enc --avif -p quality=64 -p speed=4 -o 4.avif example.png &&
heif-enc --avif -p quality=64 -p speed=5 -o 5.avif example.png &&
heif-enc --avif -p quality=64 -p speed=6 -o 6.avif example.png &&
heif-enc --avif -p quality=64 -p speed=7 -o 7.avif example.png &&
heif-enc --avif -p quality=64 -p speed=8 -o 8.avif example.png &&
heif-enc --avif -p quality=64 -p speed=9 -o 9.avif example.png
```

Observe the bug: PNG to AVIF results in larger file size the slower the speed of the AVIF encoding:

```sh

$ ls -lah
353K 0.avif
349K 1.avif
349K 2.avif
350K 3.avif
348K 4.avif
348K 5.avif
345K 6.avif
327K 7.avif
328K 8.avif
334K 9.avif

```

[Olzhas Alexandrov]

Sorry, I cannot edit the initial post to add a note:
- this issue happens for at least 20% of PNG files, but never when converting JPEG; based on a testing of about ~5000 JPEG and ~5000 PNG files

[James Zern]

On Wed, Oct 26, 2022 at 9:51 PM Olzhas Alexandrov <olzhas.a...@gmail.com> wrote:

Sorry, I cannot edit the initial post to add a note:
- this issue happens for at least 20% of PNG files, but never when converting JPEG; based on a testing of about ~5000 JPEG and ~5000 PNG files

That may have to do with the content of the files.

What's missing in this comparison is the effect on quality measured by metrics like psnr and ssim since you're compressing the avif lossily. Can you provide that data for a few images?

 

--
You received this message because you are subscribed to the Google Groups "AV1 Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to av1-discuss...@aomedia.org.
To view this discussion on the web visit https://groups.google.com/a/aomedia.org/d/msgid/av1-discuss/936a6a4b-1b1a-4b10-b77a-2f2ff070b59an%40aomedia.org.

[Olzhas Alexandrov]

Thank you for the reply.

If you remember the CLI commands to run against failing images, could you please share them?

- I'll educate myself on this topic in the meantime and then would attempt to get this data :)

It would be great to find out the root cause, as imho, it's the responsibility of a codec to at least keep the size on the same level behind the scenes, but not to increase it with bigger effort (slower speed).

[Frank Galligan]

Hi Olzhas,

So in general I try to think of speed like this "lower speed equals more encoding time, which gives better compression ratio." Compression ratio is affected by both file size and quality as James has mentioned. We usually use BD-Rate to measure this. 

Thanks,

Frank

To view this discussion on the web visit https://groups.google.com/a/aomedia.org/d/msgid/av1-discuss/a4d1c060-74ce-4dd0-bdc3-ae7bd2f6d02bn%40aomedia.org.

[Olzhas Alexandrov]

Hello Frank, let me please clarify:

• based on a testing of conversion of ~10,000 images (JPEG & PNG) to AVIF, about 20% of PNG files experience a problem:
  • the slower the speed (more effort) the worse result (bigger file size)

There's a little possibility the issue is not with `libaom`, but with `libheif`.
However, I don't know how to narrow down the issue for you.
When I do:
aomenc -p 1 -w 1534 -h 2000 --cpu-used=0 -o 0.avif example.png

I get:
webmenc> Segment::Finalize failed.
Fatal: WebM writer finalization failed.

So I used `libheif` in the initial message.
`libheif` does use `libaom` when doing encoding to AVIF, verified it as described here

[Olzhas Alexandrov]

I'm glad we both agree according to your vision, but libaom operates in the opposite (unfortunately invalid/buggy) way.

> So in general I try to think of speed like this "lower speed equals more encoding time, which gives better compression ratio."

[Olzhas Alexandrov]

Is there any way I could get any interest to this issue?
Maybe, could you please help to attempt to narrow down the issue?

When I try to narrow down the issue by using libaom directly, by running:

aomenc -p 1 -w 1534 -h 2000 --cpu-used=0 -o 0.avif example.png

I get:
webmenc> Segment::Finalize failed.
Fatal: WebM writer finalization failed.

[Cheng Chen]

Hi Olzhas,

libaom can not take .png format as input.

Instead, it takes .y4m format.

You can convert it by "ffmpeg -i example.png -pix_fmt yuv420p example.y4m"

Then "aomenc -p 1 -w 1534 -h 2000 --cpu-used=0 -o 0.avif example.y4m" shows the file size is only 32K.

Regards,

Cheng

To view this discussion on the web visit https://groups.google.com/a/aomedia.org/d/msgid/av1-discuss/ca1d89e0-a956-4ea2-b0a3-34fbcba3c86bn%40aomedia.org.

[Olzhas Alexandrov]

Thank you very much Cheng for the guidance and useful info.
I ran the exact same CLI commands you provided and unfortunately, it confirms the issue I'm raising with libaom.

Here's how you could confirm the invalid behavior in libaom:

1. As you guided, convert PNG to YUV4Mpeg

ffmpeg -i example.png -pix_fmt yuv420p example.y4m

2. And then convert Y4M to AVIF with libaom (sequentially to avoid possible issues with parallel processing):

aomenc -p 1 -w 1534 -h 2000 --cpu-used=0 -o 0.avif example.y4m &&
aomenc -p 1 -w 1534 -h 2000 --cpu-used=1 -o 1.avif example.y4m &&
aomenc -p 1 -w 1534 -h 2000 --cpu-used=2 -o 2.avif example.y4m &&
aomenc -p 1 -w 1534 -h 2000 --cpu-used=3 -o 3.avif example.y4m &&
aomenc -p 1 -w 1534 -h 2000 --cpu-used=4 -o 4.avif example.y4m &&
aomenc -p 1 -w 1534 -h 2000 --cpu-used=5 -o 5.avif example.y4m &&
aomenc -p 1 -w 1534 -h 2000 --cpu-used=6 -o 6.avif example.y4m &&
aomenc -p 1 -w 1534 -h 2000 --cpu-used=7 -o 7.avif example.y4m &&
aomenc -p 1 -w 1534 -h 2000 --cpu-used=8 -o 8.avif example.y4m &&
aomenc -p 1 -w 1534 -h 2000 --cpu-used=9 -o 9.avif example.y4m

3. Then, observe the bug (the quicker libaom processed, the smaller the files are):

ls -lah

outputs:

31K 0.avif
31K 1.avif
31K 2.avif
30K 3.avif
29K 4.avif
29K 5.avif
29K 6.avif
29K 7.avif
29K 8.avif
29K 9.avif

[James Zern]

Hi,

On Thu, Nov 3, 2022 at 7:33 PM Olzhas Alexandrov <olzhas.a...@gmail.com> wrote:

Thank you very much Cheng for the guidance and useful info.
I ran the exact same CLI commands you provided and unfortunately, it confirms the issue I'm raising with libaom.

Here's how you could confirm the invalid behavior in libaom:

1. As you guided, convert PNG to YUV4Mpeg

ffmpeg -i example.png -pix_fmt yuv420p example.y4m

2. And then convert Y4M to AVIF with libaom (sequentially to avoid possible issues with parallel processing):

aomenc -p 1 -w 1534 -h 2000 --cpu-used=0 -o 0.avif example.y4m &&
aomenc -p 1 -w 1534 -h 2000 --cpu-used=1 -o 1.avif example.y4m &&
aomenc -p 1 -w 1534 -h 2000 --cpu-used=2 -o 2.avif example.y4m &&
aomenc -p 1 -w 1534 -h 2000 --cpu-used=3 -o 3.avif example.y4m &&
aomenc -p 1 -w 1534 -h 2000 --cpu-used=4 -o 4.avif example.y4m &&
aomenc -p 1 -w 1534 -h 2000 --cpu-used=5 -o 5.avif example.y4m &&
aomenc -p 1 -w 1534 -h 2000 --cpu-used=6 -o 6.avif example.y4m &&
aomenc -p 1 -w 1534 -h 2000 --cpu-used=7 -o 7.avif example.y4m &&
aomenc -p 1 -w 1534 -h 2000 --cpu-used=8 -o 8.avif example.y4m &&
aomenc -p 1 -w 1534 -h 2000 --cpu-used=9 -o 9.avif example.y4m

3. Then, observe the bug (the quicker libaom processed, the smaller the files are):

Can you add the output from the --psnr flag for those files, please?

For still images --allintra may be a better choice, along with setting a --cq-level value.

[Olzhas Alexandrov]

> Can you add the output from the --psnr flag for those files, please?

> For still images --allintra may be a better choice, along with setting a --cq-level value.

Hello jz...@google.com,

I attempted to do what you mentioned on the following CLI command (0 is replaced by 0-9; i.e. ran 10 times):

CPU_USED=0 && aomenc -p 1 -w 1534 -h 2000 --cpu-used=$CPU_USED -o $CPU_USED.avif example.y4m

and here are the results:

1. adding at the end of the CLI cmd only: --allintra

    same bug could be observed as reported in all previous messages above (AND extreme increase in average size)

CPU_USED=0 576904B 4615232b/f 115380800b/s   44576 ms (0.02 fps)
CPU_USED=1 572585B 4580680b/f 114517000b/s   26351 ms (0.04 fps)
CPU_USED=2 572781B 4582248b/f 114556200b/s   21023 ms (0.05 fps)
CPU_USED=3 573729B 4589832b/f 114745800b/s   13339 ms (0.07 fps)
CPU_USED=4 566657B 4533256b/f 113331400b/s 6000301 us (0.17 fps)
CPU_USED=5 567052B 4536416b/f 113410400b/s 4746307 us (0.21 fps)
CPU_USED=6 560871B 4486968b/f 112174200b/s  916281 us (1.09 fps)
CPU_USED=7 546925B 4375400b/f 109385000b/s  612041 us (1.63 fps)
CPU_USED=8 551386B 4411088b/f 110277200b/s  362611 us (2.76 fps)
CPU_USED=9 566578B 4532624b/f 113315600b/s  101124 us (9.89 fps)

2. adding at the end of the CLI cmd only: --allintra --cq-level=40

    same bug could be observed as reported in all previous messages above

CPU_USED=0 115493B  923944b/f 23098600b/s   39267 ms (0.03 fps)
CPU_USED=1 112624B  900992b/f 22524800b/s   15328 ms (0.07 fps)
CPU_USED=2 111785B  894280b/f 22357000b/s   12251 ms (0.08 fps)
CPU_USED=3 111476B  891808b/f 22295200b/s 8205609 us (0.12 fps)
CPU_USED=4 108706B  869648b/f 21741200b/s 4037591 us (0.25 fps)
CPU_USED=5 108506B  868048b/f 21701200b/s 2800509 us (0.36 fps)
CPU_USED=6 104907B  839256b/f 20981400b/s  786029 us (1.27 fps)
CPU_USED=7  96859B  774872b/f 19371800b/s  366507 us (2.73 fps)
CPU_USED=8  93157B  745256b/f 18631400b/s  125336 us (7.98 fps)
CPU_USED=9  90832B  726656b/f 18166400b/s   44011 us (22.72 fps)

3. adding at the end of the CLI cmd only: --allintra --cq-level=40 --psnr

    same bug could be observed as reported in all previous messages above

CPU_USED=0 115493B  923944b/f 23098600b/s   39364 ms (0.03 fps)
Stream 0 PSNR (Overall/Avg/Y/U/V) 36.312 36.312 34.567 66.439 52.850 23098600 bps   39364 ms

CPU_USED=1 112624B  900992b/f 22524800b/s   15366 ms (0.07 fps)
Stream 0 PSNR (Overall/Avg/Y/U/V) 36.153 36.153 34.411 66.439 52.195 22524800 bps   15366 ms

CPU_USED=2 111785B  894280b/f 22357000b/s   12445 ms (0.08 fps)
Stream 0 PSNR (Overall/Avg/Y/U/V) 36.124 36.124 34.375 66.439 54.124 22357000 bps   12445 ms

CPU_USED=3 111476B  891808b/f 22295200b/s 8476690 us (0.12 fps)
Stream 0 PSNR (Overall/Avg/Y/U/V) 36.068 36.068 34.329 66.439 51.455 22295200 bps    8476 ms

CPU_USED=4 108706B  869648b/f 21741200b/s 4088714 us (0.24 fps)
Stream 0 PSNR (Overall/Avg/Y/U/V) 35.828 35.828 34.093 66.439 50.414 21741200 bps    4088 ms

CPU_USED=5 108506B  868048b/f 21701200b/s 2762357 us (0.36 fps)
Stream 0 PSNR (Overall/Avg/Y/U/V) 35.676 35.676 33.939 66.439 50.602 21701200 bps    2762 ms

CPU_USED=6 104907B  839256b/f 20981400b/s  773993 us (1.29 fps)
Stream 0 PSNR (Overall/Avg/Y/U/V) 35.356 35.356 33.625 66.439 49.278 20981400 bps     773 ms

CPU_USED=7  96859B  774872b/f 19371800b/s  364323 us (2.74 fps)
Stream 0 PSNR (Overall/Avg/Y/U/V) 34.985 34.985 33.259 66.439 48.262 19371800 bps     364 ms

CPU_USED=8  93157B  745256b/f 18631400b/s  127400 us (7.85 fps)
Stream 0 PSNR (Overall/Avg/Y/U/V) 34.592 34.592 32.851 66.439 50.471 18631400 bps     127 ms

CPU_USED=9  90832B  726656b/f 18166400b/s   42947 us (23.28 fps)
Stream 0 PSNR (Overall/Avg/Y/U/V) 34.222 34.222 32.481 66.439 49.910 18166400 bps      42 ms

[James Zern]

You can see the overall PSNR (which measures the output quality) trends downward as the speed is increased, so I think this is working as expected. Each speed level may select different tools causing the quality and size to fluctuate while attempting to hit the target data rate or quality.

[Olzhas Alexandrov]

What you are saying, jz...@google.com is:

> I think [ cpu-used parameter should affect quality and it ] is working as expected

I have an opposite view: cpu-used should not affect quality.
If cpu-used must affect quality because you cannot make it work otherwise, then you should:

• at least change the description of the parameter to indicate cpu-used affects the quality
• or change the name of the parameter to something that might; like effort instead of cpu-used

But I still hope you would consider changing program behavior:

• to make only cq-level and other parameters that have any reasonable correlation with quality (ex. cpu-used doesn't have) to affect quality

[Jingning Han]

Here is my understanding of the tradeoff between encoder complexity and the reconstruction quality.

When the quantization parameter is moderate to high, many residual signals in the transform domain will be quantized to zero.

In this setting, the prediction quality translates into the reconstruction quality. Hence a higher complexity mode that searches more prediction modes is likely to give a better reconstruction quality.

Now when the quantization parameter is small, it enters the high resolution quantization era. This is the setting where encoder complexity doesn't affect the reconstruction quality, since that's mainly decided by the quantization step size.

Instead the encoder complexity usually impacts the resulting file size, as a better predicted block tends to consume less bits for the same reconstruction quality.

To view this discussion on the web visit https://groups.google.com/a/aomedia.org/d/msgid/av1-discuss/4e7d45e0-b8f8-4d99-a0bb-697815db4f5fn%40aomedia.org.

[Olzhas Alexandrov]

Jing, the parameter we are discussing is called CPU USED.
--cpu-used has nothing to do in any programming dictionary with the encoding complexity.

99% of developers would expect --cpu-used to affect only the speed of the process, not the resulting output.

[James Zern]

On Fri, Nov 4, 2022 at 11:23 PM Olzhas Alexandrov <olzhas.a...@gmail.com> wrote:

Jing, the parameter we are discussing is called CPU USED.
--cpu-used has nothing to do in any programming dictionary with the encoding complexity.

99% of developers would expect --cpu-used to affect only the speed of the process, not the resulting output.

With lossless encoding I agree with your expectations about file size versus the effort to encode. As Jingning explains the process is a bit different when doing lossy encodes.