The WebP comparative "study"
Thomas Richter
with some surprise I was able to find the quoted "study" on the WebP
technology here:
http://code.google.com/intl/de-DE/speed/webp/docs/c_study.html
I'm not clear whom to write as the author of the study remains unknown,
sorry if that's not you. I would believe that
a good study should quote who did it - it is good scientific practice to
provide a contact author.
Unfortunately, this study seems to be done pretty careless, to put it
mildly - there are good reasons why JPEG uses
established procedures to test image compression codecs. Just to comment
some:
o) To compare image compression codecs, you compare quality vs. rate.
Quality is defined in terms of image
quality indices, of which PSNR is just one (and probably not even a good
one), rate is in bits per pixel. This gives curves
that show the quality/rate tradeoff between codecs. No such curves are
given, but rather curves showing "maximum
compression" - what should that mean for a lossy compression, actually?
Under lossless coding? It would astonish me if you
would really mean the lossless mode of JPEG-1 - that has rarely been
deployed outside of the medical industry, and the IJG
code ("libjpeg"), for example, does not support it.
o) If you measure codecs, make sure you provide full parameters and
information of what you actually did. For example,
kakadu 6.4 in its default setting is *not* PSNR optimal, but visually
optimal, i.e. it employs so called CSF weights. So don't be surprised
that the PSNR for kakadu seems to be lower that that of a second codec -
the images should look better, though. Question
is - what is "better" ?
o) What is an "average PSNR", actually? Why does averaging PSNR makes
sense, and under which terms has this averaging been
done? Has a rate limit be defined? Unfortunately, I don't seem to see
that mentioned anywhere.
o) What is an average compression percentage? The image compression
community measures compression in bits per pixel, not
in percent (nit-picking, of course), but even then, I wonder what
exactly happened here.
o) Selection of test images. Google seem to have used average images
from the web - or maybe I'm wrong?
Most of such images have undergone compression
already, and such show an image statistics that is non-natural and
typical for the compressor used in first place. Thus, if you compress
a JPEG image with JPEG 2000, yes , it will show a sub-optimal result.
The same goes for the reverse, of course. If you want to measure
image compression performance, it is of utter importance to use original
uncompressed images, and not just arbitrary images that
underwent some unknown compression already. The JPEG provides a test
image set, for example, that is used by the committee for
such purposes.
The only result of a study using already compressed images is that a
codec whose technology is close to the original technology used to
arrive at the test image will perform better - and that would be the
JPEG-1 standard.
o) It might or might not be known, but the JPEG is actually open for any
proposals - in fact, there is currently an open proposal for Advanced
Image Coding. Such a proposal means that contributors may bring in code,
and such code is then evaluated carefully, under the same strict
guidelines as any other code is and has been evaluated. Of course,
google technology is as welcome as any other technology.
To give you some idea what is wrong with this comparison, here is an
unbiased first, very quick test that made me very feel very
suspicious about this study:
Compress a typical test image with the webp code, for example "boats"
(or shipbig in my collection - s.bmp):
thor@cutter:~/src/webp/libwebp> webpconv -quality 80 s.bmp
Target quality 80
processing s.bmp
Output file s.webp
thor@cutter:~/src/webp/libwebp> mv s.bmp so.bmp
thor@cutter:~/src/webp/libwebp> webpconv -format bmp s.webp
Output format bmp
processing s.webp
Output file s.bmp
thor@cutter:~/src/webp/libwebp> difftest_ng so.bmp s.bmp
PSNR: 33.5615
thor@cutter:~/src/webp/libwebp> ls -l s.webp
-rw-r--r-- 1 thor thor 165428 1. Okt 23:40 s.webp
Thus, we have a PSNR of 33.56 at a bitrate of 165428 bytes. Now compress
the same with JPEG2000, using PSNR optimal settings
(I'm not doing anything fancy here):
thor@cutter:~/src/webp/libwebp> j2k -by 165428 so.bmp s.j2k
thor@cutter:~/src/webp/libwebp> ls -l s.j2k
-rw-r--r-- 1 thor thor 165497 1. Okt 23:42 s.j2k
A close match, I would say. Reconstruction:
thor@cutter:~/src/webp/libwebp> j2k s.j2k s2.bmp
thor@cutter:~/src/webp/libwebp> difftest_ng so.bmp s2.bmp
PSNR: 34.1319
That makes a PSNR difference of 0.6dB, which is quite a lot in the image
compression community. I'm not stating that this
reflects the total performance of WebP, it is only a very quick, very
early, very simple test. But what I find shocking that
nothing in this "study" actually reflects such an outcome. It is the
very *first* thing that should be done.
Test software used here is the 0.8 source code I downloaded and compiled
on a Debian 64 bit machine, and a JPEG 2000 implementation
from my own hands, but you can surely verify similar results using the
reference software, the JJ2000 or the jasper. Options, details
and performance may differ a bit, but that is nothing that would explain
the missing 0.6 dB. Test image for this check is, as said,
the "boats" image from the waterloo test set, also part of the (old)
JPEG test set. It is only used because it is the first image I
had available, and it is traditionally the first image I test with - no
other reason. (Yes, I should test with lena - but that's a too small,
too bad quality image to be of much use).
Best regards,
Thomas Richter