Experimental zstd-PNG converter

[Takayuki Matsuoka]

Hi,

I've written experimental program which encodes PNG to PNG variant and vice versa.
https://gist.github.com/t-mat/03d3a6782a60bfef8239

This PNG variant (.zspng) uses zstd for its IDAT compression instead of the Deflate (zlib).
Result looks good. There are some quirks (frymire.png, serrano.png), but overall results are almost same as Deflate's.

[Yann Collet]

That's great Takayuki !

I've been looking into doing something equivalent for some time,

I'm impressed you did it in such a short source file !

One question I had in mind :

Is the result solely attributable to FSE, or is the LZ part any useful for such a scenario.

My suspicion is that the LZ part is useful for Huffman, as a way to break the 1 bit/symbol limit.

But with FSE as back end, it's less clear if that's really useful.

Regards

[Takayuki Matsuoka]

> Is the result solely attributable to FSE, or is the LZ part any useful for such a scenario.

I've added pure FSE variant to my example : https://gist.github.com/t-mat/745c16cfa2b9e387ea16

> My suspicion is that the LZ part is useful for Huffman, as a way to break the 1 bit/symbol limit.
> But with FSE as back end, it's less clear if that's really useful.

Basically, your intuition is right. But it seems still useful for complex but highly compressible datas.

[Yann Collet]

Thanks Takayuki

Your results are very clear.

So LZ layer always help, even if just a little.

Therefore, better keep it.

There are 2 cases in which it helps a lot, frymire.png and serrano.png.

I note that for these 2 samples, zstd results are not very good compared to deflate.

I guess that's because the fast version only looks for matches of 7+ bytes.

This is probably too much for these samples.

I therefore suspect the HC version will fare much better for these cases, since it will go down to 4+ bytes,

This version remains to be completed...

[Frédéric Kayser]

Hello,
Interesting tool but the images picked for testing have some caveats:
- most of them (all expected frymire and serrano) are much more suited for lossy compression and not lossless.
- the PNG files have not been optimized, I know that ZSTD is still in its infancy and putting it against top notch PNG compressors may not be fair but if you want to make a point confront your strongest opponent not the weakest one.
I've optimized the PNGs with the following tools: PNGOUT, ZopfliPNG and Defluff.
Once only PNGOUT followed by Defluff (when PNGOUT default filter 5 could not reduce the file size filter 0 was used), since last summer ZopfliPNG has become one of the best PNG optimization tool therefore I have also used it on the previously optimized files (and again Defluff to shove some more byte) options used: zopflipng --iterations=20 --splitting=3 --filters=01234mepb

The resulting file size:
418873 airplane.png
236624 arctichare.png
624510 baboon.png
176925 barbara.png
165835 boat.png
634836 cat.png
446739 fruits.png
225637 frymire.png
569739 girl.png
159042 goldhill.png
473057 lena.png
607758 monarch.png
504199 peppers.png
159874 pool.png
754110 sails.png
 95670 serrano.png
678960 tulips.png
615000 watch.png
138151 zelda.png

If you run these new PNGs through your tool you'll probably find different resulting files since for most of them PNG filtering has changed and thus the raw data to compress is now different.
Archive containing the optimized files here: http://frdx.free.fr/better-pngs.zip

I will get back to you with a set of PNGs that will be probably more representative of what's really stored in .png format on the Web.

Regards
--
Frédéric Kayser

Le vendredi 6 février 2015 07:19:22 UTC+1, Takayuki Matsuoka a écrit :

[Takayuki Matsuoka]

Hi Frédéric,

Thanks.  I didn't notice anything about these points.
And I'm looking forward to seeing your result !

[Frédéric Kayser]

Hello,
could you rerun your tests using the 8 PNGs from this archive: http://frdx.free.fr/real-pngs.zip and publish the results? Sample 5 and 8 should benefit from ZSTD larger search window (512K vs 32K for Deflate), sample 5 had been especially crafted to illustrate the search window limit, explanations here.

(I still have to compile ZSTD and your tool to really start studying how it reacts to PNG data, unfortunately I've not a lot of free time for this).

Regards
--
Frédéric Kayser

[Takayuki Matsuoka]

Thanks !
Revamped : https://gist.github.com/t-mat/745c16cfa2b9e387ea16