state of lossless

153 views
Skip to first unread message

Pascal Massimino

unread,
Apr 11, 2012, 5:01:25 AM4/11/12
to WebP Discussion
Hi everybody,

you might have noticed a big transfer and refresh of files from 'experimental' to the 'master' branch
in the repository (here). It's mainly the new refreshed code for lossless format! It's getting fairly close
to getting usable now and comments about the format, code, etc. is much welcome.
Here's a summary of the updates:
 * a reviewable PDF for the lossless spec is available as doc/webp-lossless-format-spec.pdf from experimental branch.
   Please, have a look and comment! There's surely some details to iron out regarding robustness of the syntax, ease-of-implementation, corner cases, etc.
 * expect a corresponding document for the alpha in lossy-format soon. It will describe the code in src/utils/alpha.c mainly.
 * a lossless decoder that is fully and transparently integrated in the main libwebp library. You can keep calling WebPDecode() on lossless files and it should work transparently, e.g. Rescaling and cropping also work, note. The decoder's main code starts at src/dec/vp8l.c.
 * an updated and sped-up encoder in the experimental branch, under src/lossless/enc/... This encoder abides to the new bitstream syntax.

So, long story short, if you want to try the new lossless, it's basically:

git checkout experimental
make -f makefile.unix
cd src/lossless
make -f makefile.unix
tools/png2webpll input.png -o output.webp
../../examples/dwebp output.webp -o output.png

feedback, patches, timing measurement, etc. are much welcome.
enjoy!
skal

James Zern

unread,
Apr 11, 2012, 11:07:27 PM4/11/12
to WebP Discussion


On Apr 11, 2:01 am, Pascal Massimino <pascal.massim...@gmail.com>
wrote:
> Hi everybody,
>
> you might have noticed a big transfer and refresh of files from
> 'experimental' to the 'master' branch
> in the repository
> (here<https://gerrit.chromium.org/gerrit/#q,status:merged+project:webm/libw...>).
> It's mainly the new refreshed code for lossless format! It's getting fairly
> close
> to getting usable now and comments about the format, code, etc. is much
> welcome.

I refreshed the executables in the download section [1] if anyone
wants to try the update without having to build from source.

[1] http://code.google.com/p/webp/downloads/list

Szabolcs Péter

unread,
Apr 12, 2012, 12:59:51 PM4/12/12
to webp-d...@webmproject.org
Hmmm, the new code sometimes creates bigger files than the old (even with -c 100), sometimes smaller, though mostly about the same size... 

However, it's an order of magnitude faster, and doesn't crash on some odd files, very nice work indeed! :)

Cheers,
 SyP

Nigel Tao

unread,
Apr 12, 2012, 11:51:51 PM4/12/12
to webp-d...@webmproject.org
On 11 April 2012 19:01, Pascal Massimino <pascal.m...@gmail.com> wrote:
>  * a reviewable PDF for the lossless spec is available
> as doc/webp-lossless-format-spec.pdf from experimental branch.
>    Please, have a look and comment! There's surely some details to iron out
> regarding robustness of the syntax, ease-of-implementation, corner cases,
> etc.

Some notes from a quick skim:

The ARGB values are straight alpha (like PNG), not pre-multiplied
alpha, right? I'm guessing so, but I don't see that in the spec.

Section 4, "Colors are looked up by indexing them by (0x1e35a7bd * color)"
Do you need to specify endianness for how to convert an ARGB to a uint32?

Section 6, "<lz77-coded image> ::= (<rgba-pixel> etc"
s/rgba/argb/, right?

Jyrki Alakuijala

unread,
Apr 13, 2012, 4:56:12 AM4/13/12
to webp-d...@webmproject.org
On Fri, Apr 13, 2012 at 5:51 AM, Nigel Tao <nige...@golang.org> wrote:
> On 11 April 2012 19:01, Pascal Massimino <pascal.m...@gmail.com> wrote:
>>  * a reviewable PDF for the lossless spec is available
>> as doc/webp-lossless-format-spec.pdf from experimental branch.
>>    Please, have a look and comment! There's surely some details to iron out
>> regarding robustness of the syntax, ease-of-implementation, corner cases,
>> etc.
>
> Some notes from a quick skim:
>
> The ARGB values are straight alpha (like PNG), not pre-multiplied
> alpha, right? I'm guessing so, but I don't see that in the spec.
>
> Section 4, "Colors are looked up by indexing them by (0x1e35a7bd * color)"
> Do you need to specify endianness for how to convert an ARGB to a uint32?

no, the argb has been defined as a uint32 in the spec, defining that b
is the lowest 8 bits

>
> Section 6, "<lz77-coded image> ::= (<rgba-pixel> etc"
> s/rgba/argb/, right?

fixing this

> --
> You received this message because you are subscribed to the Google Groups "WebP Discussion" group.
> To post to this group, send email to webp-d...@webmproject.org.
> To unsubscribe from this group, send email to webp-discuss...@webmproject.org.
> For more options, visit this group at http://groups.google.com/a/webmproject.org/group/webp-discuss/?hl=en.
>

Jyrki Alakuijala

unread,
Apr 13, 2012, 5:21:30 AM4/13/12
to webp-d...@webmproject.org
2012/4/12 Szabolcs Péter <syp...@gmail.com>:

> Hmmm, the new code sometimes creates bigger files than the old (even with -c
> 100), sometimes smaller, though mostly about the same size...

On our 1000 image test corpus, the new code compresses about 50x
faster than the encoder in the first release in November, but produces
about 3 % more bytes. Most of the compression degradation is because
of algorithmic speedups (possibly around 2 %), and the rest because of
simplification of the format (1 %).

Encoding at -c 0 seems now faster than libpng and produces still 25 %
less bytes, and -c 95 (default) is a bit faster than pngout and gives
similarly 25 % less bytes.

Another piece of good news is that the decoding speed is also a bit
faster than libpng now, although so far I have only benchmarked the
experimental decoder (the production decoder should end up having
similar or better performance).

Jyrki Alakuijala

unread,
Apr 13, 2012, 5:47:55 AM4/13/12
to webp-d...@webmproject.org
On Fri, Apr 13, 2012 at 5:51 AM, Nigel Tao <nige...@golang.org> wrote:
> On 11 April 2012 19:01, Pascal Massimino <pascal.m...@gmail.com> wrote:
>>  * a reviewable PDF for the lossless spec is available
>> as doc/webp-lossless-format-spec.pdf from experimental branch.
>>    Please, have a look and comment! There's surely some details to iron out
>> regarding robustness of the syntax, ease-of-implementation, corner cases,
>> etc.
>
> Some notes from a quick skim:
>
> The ARGB values are straight alpha (like PNG), not pre-multiplied
> alpha, right? I'm guessing so, but I don't see that in the spec.

I thought very carefully about this in the beginning of the work,
guided by earlier experiments in trying out byte-size-optimize storing
of pre-multiplied argb images in png.

I decided to not to specifically support the pre-multiplied mode. The
reasons for it are relatively complicated to explain, but I am giving
it a try.
1. Doing the pre-multiplication immediately after loading the image is
a cheap operation.
2. Alpha is typically not strongly correlated with R,G,B, which means
that pre-multiplication mixes independent entropy sources, increasing
the total entropy in R,G, and B channels.
3. Alpha-premultiplication throws away information so it should be, in
theory, good for compression, but one would need to have delicate
compensation algorithms to decorrelate the A from the R,G,B signals
for best compression. Such mechanisms would increase the complexity of
the format.
4. Pre-processing of the R,G,B channels can be done in a way that
throws away same amount of information than pre-multiplication, but
does no increase the entropy of R,G,B channels.
5. We need to keep a non-premultiplicative mode for other uses and for
compatibility with existing formats (for example png).
6. Having a bit to indicate premultiplication would complicate the
format, and decoder and encoder implementations.

The second statement of the abstract-section states the following, but
this message is not confirmed in the actual text.

"The lossless format stores and restores the pixel values exactly,
including the color values for zero alpha pixels."

> Section 4, "Colors are looked up by indexing them by (0x1e35a7bd * color)"
> Do you need to specify endianness for how to convert an ARGB to a uint32?
>
> Section 6, "<lz77-coded image> ::= (<rgba-pixel> etc"
> s/rgba/argb/, right?
>

Frédéric Kayser - CryoPNG

unread,
May 12, 2012, 12:07:41 AM5/12/12
to WebP Discussion
On 13 avr, 11:47, Jyrki Alakuijala <jy...@google.com> wrote:

> 4. Pre-processing of the R,G,B channels can be done in a way that
> throws away same amount of information than pre-multiplication, but
> does no increase the entropy of R,G,B channels.

> "The lossless format stores and restores the pixel values exactly,
> including the color values for zero alpha pixels."

Hello,
I'm the author of the hack called "CryoPNG" a tool intended to clean
the RGB components of completely transparent pixels in PNG files.
The PNG specs stated:
"If a region is completely transparent, don't zero out the underlying
color pixels in order to improve compression unless you've notified
the user in some way."

Well, I did worse since CryoPNG actually zeroes out the output of the
line filters, that means completely transparent pixel can end up with
non zero RGB components (the value of the reverse transformation of
the filter indeed) those will be zeroed once filtered, thus the
deflate engine receives more zeros, this slightly improves compression
of RGBA PNG images.

Now the funny thing is that at least some CryoPNG optimized PNG files
end up producing smaller webpll files:
https://extrememoderate.wordpress.com/2011/11/21/webp-gallery-oddities/

My question is:
What would be your stance if I wrote a tool that does the same but
specifically tuned for webpll files?

Pascal Massimino

unread,
May 13, 2012, 7:42:26 PM5/13/12
to webp-d...@webmproject.org
Hi Frédéric,

there's a -alpha_cleanup option to cwebp that i introduced once:
which is kind of attempting to cleanup zero-alpha area.
I could really make it work consistently well.
The function for doing the cleanup is in enc/picture.c: WebPCleanupTransparentArea
The problem is that the vp8 entropy coding can be counter-intuitive and adding
zeros is not always the optimal solution, when you consider all the prediction and filtering
that is happening underneath.

Glad if you could have a look and improve it!

skal
Reply all
Reply to author
Forward
0 new messages