A truly comparative study

19 views

Skip to first unread message

Thomas Richter

unread,

Oct 3, 2010, 4:48:39 PM10/3/10

to webp-d...@webmproject.org

After having seen google's(?) "comparative study", I just took the time
to perform a somewhat more scientific
study of the webp algorithm of which I want to share the results here.

Participating codecs are the IJG JPEG codec, a JPEG2000 implementation,
a JPEG XR implementation and
the webp implementation.

Test images aren't "arbitrary images" from the web, but rather images
from the JPEG test corpus. Images are,
specifically, uncompressed, and haven't been compressed before. They are
photographic in nature, differing
in their features.

What has been measured: PSNR (logarithm of the mean square error), the
log of the peak error (PRE, l^infinity error),
and multiscale mean-ssim. None of these quality indices can cover all
image defects, and I will provide additional
outcomes from my own subjective inspection. It should be noted that this
is not a ITU-conforming subjective quality
evaluation as it consists only of a single expert observer, and my
monitor is not calibrated, nor am I an unbiased observer,
I've seen too many compressed images, thus consider me an expert viewer.

Subjective impression:

The positive:

webp performs pretty well on images with flat background and sharp, easy
to identify edges. It preserves such
edges pretty well and there avoids ringing (Gibbs artefacts) pretty
well, much better than JPEG 2000. This means
that, for example, the "bike" comes out pretty well. Quite unlike JPEG,
bocking is not quite that obvious.

For images like bike, SSIM scores are pretty high, in the
high-compression area higher than JPEG 2000.

It should be noted that this test was using a PSNR-optimized JPEG 2000,
a visually optimized JPEG 2000 would have shown a
much better performance.

The negative:

Due to its block-based nature, it cannot preserve low-contrasts and
smooth details; while blocking isn't as visible as in JPEG,
it is still pretty annoying (e.g. the sky in the cafe-image contains
anoying block artifacts below 0.7 bpp, the fruit in the bike
image are blocky with clearly visible blocking defects that are a bit
smoothed out, but still visible).

Since it seems to run some kind of edge-detection with selective
sharpening, considerable loss can be observed in textured
image areas, surprisingly even more than JPEG 2000 which has a known
problem with such areas. Such structure loss is
mostly visible in the "honolulu image test set" where structure loss
happens in the grass, the trees, the plants etc, where it
is pretty annoying for compression ratios up to about 0.7 bpp.

In the high-quality domain, the performance is quite bad, i.e. PSNR
values are pretty low, and often below that of JPEG XR,
which is again below that of JPEG 2000, probably a quality loss a
professional photographer would not want to tolerate.
In this high quality domain, JPEG 2000 dominates in PSNR performance,
followed by XR. WebP drops below XR.

This is probably because WebP is based on a video codec, and in the
video domain much lower qualities are acceptable than
in the still image domain; missing structure is there often masked by
motion and thus not quite as relevant as in the
still image sector.

Should google consider webp to become a dominant still image codec, then
it still has quite a long road to go. Especially, it
must leave its roots from the video domain behind; video and still image
compression are quite different with very different
demands. The smoothing and edge-preserving features of webP are probably
precisely what the video domain needs, but
for still images such qualities are not acceptable and more work needs
to go into the lower compression domain.

Specifically, I would be interested in the features the code has to
offer. Can I tune it to a specific quality index? Does it have
a reasonable rate-allocation? Quite like JPEG-XR, unfortunately, its
design seems to not follow quite the state of the art,
which is too bad.

Now, some numbers. If you want test images or pdf plots from these
numbers, please write. The mail is valid.

Images from the "Honolulu test set" (mostly lots of structure/texture
with some edges):

Quality/Compression Comparison for Image /store/pics/core1/honolulu_zoo.ppm
Automatically generated by jpegcompare (c) 2005,2007 THOR Software,
Thomas Richter

Image Details:
TIFF Directory at offset 0x1d13e84 (30490244)
Image Width: 3906 Image Length: 2602
Bits/Sample: 8
Compression Scheme: None
Photometric Interpretation: RGB color
FillOrder: msb-to-lsb
Orientation: row 0 top, col 0 lhs
Samples/Pixel: 3
Rows/Strip: 1
Planar Configuration: single image plane
DocumentName: Standard Input
ImageDescription: converted PNM file

Codec Size PSNR SSIM PRE VDP PER PQS
mDCT-PSNR VSSIM

JPEG1(BASE) .24572 25.3497 18.3502 2.4648
JPEG XR(peg) .24811 28.5928 21.5529 5.7193
JPEG2000 .24999 29.7172 24.694 5.7193
WebP (default) .24135 28.3329 22.4205 4.8437

JPEG1(BASE) .48781 29.3783 28.2681 2.6016
JPEG XR(peg) .49520 31.3641 27.2616 8.9500
JPEG2000 .49965 32.2513 30.639 7.7070
WebP (default) .48805 30.8810 28.7934 5.8519

JPEG1(BASE) .74496 31.2041 33.7754 3.9943
JPEG XR(peg) .72623 32.9047 30.2854 10.4010
JPEG2000 .74969 33.8411 34.225 9.1430
WebP (default) .73814 32.3646 32.7754 7.7070

JPEG1(BASE) .99568 32.2086 37.22 3.6246
JPEG XR(peg) .99149 34.2599 32.9833 13.1670
JPEG2000 .99998 35.1490 35.9608 13.4829
WebP (default) .97383 33.3453 35.317 7.0692

JPEG1(BASE) 1.23947 32.9478 39.6125 5.7854
JPEG XR(peg) 1.21205 35.1852 34.7748 13.4829
JPEG2000 1.24982 36.2174 38.1799 16.5351
WebP (default) 1.19940 34.0187 37.5638 8.3954

JPEG1(BASE) 1.48881 33.5231 41.5334 7.8741
JPEG XR(peg) 1.49880 36.2051 36.6249 16.3095
JPEG2000 1.49973 37.0688 39.8126 16.5351
WebP (default) 1.47894 34.5662 39.2235 8.3954

JPEG1(BASE) 1.99485 34.5575 44.4474 9.0460
JPEG XR(peg) 1.97789 37.6637 39.1843 18.8828
JPEG2000 1.99981 38.7725 42.7462 17.7605
WebP (default) 1.78361 35.2061 41.4709 8.7611

Quality/Compression Comparison for Image
/store/pics/core1/oahu_northcoast.ppm
Automatically generated by jpegcompare (c) 2005,2007 THOR Software,
Thomas Richter

Image Details:
TIFF Directory at offset 0x1d13e84 (30490244)
Image Width: 2602 Image Length: 3906
Bits/Sample: 8
Compression Scheme: None
Photometric Interpretation: RGB color
FillOrder: msb-to-lsb
Orientation: row 0 top, col 0 lhs
Samples/Pixel: 3
Rows/Strip: 1
Planar Configuration: single image plane
DocumentName: Standard Input
ImageDescription: converted PNM file

Codec Size PSNR SSIM PRE VDP PER PQS
mDCT-PSNR VSSIM

JPEG1(BASE) .24911 29.9936 20.524 6.3327
JPEG XR(peg) .24994 33.5705 23.5757 11.1056
JPEG2000 .24988 34.3284 27.3149 10.8643
WebP (default) .23854 32.9357 23.6896 10.8643

JPEG1(BASE) .49635 34.1553 30.4389 9.2412
JPEG XR(peg) .48373 36.1475 29.6524 15.2618
JPEG2000 .49995 36.8104 34.463 13.4829
WebP (default) .49244 35.6100 30.9478 12.5678

JPEG1(BASE) .74637 36.0953 36.105 8.2181
JPEG XR(peg) .74374 37.9258 34.1224 16.7668
JPEG2000 .74933 38.3391 38.667 18.3036
WebP (default) .68683 36.7612 34.4414 12.8622

JPEG1(BASE) .97337 37.1953 39.6164 10.8643
JPEG XR(peg) .99936 39.2013 37.2814 19.5035
JPEG2000 .99988 39.5612 42.0758 18.5884
WebP (default) .91216 37.7376 38.1235 15.4614

JPEG1(BASE) 1.24114 38.1526 42.7038 11.3538
JPEG XR(peg) 1.24873 40.1951 39.736 20.5266
JPEG2000 1.24956 40.5814 44.0036 19.8313
WebP (default) 1.14927 38.4825 40.2633 16.3095

JPEG1(BASE) 1.49533 38.8432 44.8807 12.0072
JPEG XR(peg) 1.44909 40.8743 41.395 21.6864
JPEG2000 1.49997 41.4320 45.9322 22.1102
WebP (default) 1.46993 39.4053 43.5573 15.6658

JPEG1(BASE) 1.94443 39.7853 47.5739 14.6888
JPEG XR(peg) 1.86167 42.1170 44.2206 24.0484
JPEG2000 1.99991 42.8312 49.8971 22.5557
WebP (default) 1.99181 40.5636 47.1347 16.0896

ISO test images (cafe and bike): cafe contains both structure and smooth
areas with slow contrast
variations, bike contains flat backgrounds with sharp razor edges:

Quality/Compression Comparison for Image /store/pics/core1/bike.rgb.ppm
Automatically generated by jpegcompare (c) 2005,2007 THOR Software,
Thomas Richter

Image Details:
TIFF Directory at offset 0xf00008 (15728648)
Image Width: 2048 Image Length: 2560
Bits/Sample: 8
Compression Scheme: None
Photometric Interpretation: RGB color
FillOrder: msb-to-lsb
Orientation: row 0 top, col 0 lhs
Samples/Pixel: 3
Rows/Strip: 1
Planar Configuration: single image plane
DocumentName: Standard Input
ImageDescription: converted PNM file

Codec Size PSNR SSIM PRE VDP PER PQS
mDCT-PSNR VSSIM

JPEG1(BASE) .23397 23.3513 17.4265 0.2418
JPEG XR(peg) .24093 26.9511 19.8221 5.0850
JPEG2000 .24988 28.3080 21.9744 4.6090
WebP (default) .24929 28.3101 22.7754 5.0241

JPEG1(BASE) .49014 28.1556 25.9838 3.6246
JPEG XR(peg) .49496 30.4495 25.0685 8.6682
JPEG2000 .49984 31.4741 27.14 7.9588
WebP (default) .48011 31.2807 27.6972 7.7070

JPEG1(BASE) .73962 30.2826 30.4796 6.7671
JPEG XR(peg) .72379 32.3479 28.1214 10.9842
JPEG2000 .74974 33.4409 30.3134 9.9611
WebP (default) .74317 33.1729 31.5674 8.7611

JPEG1(BASE) .98588 31.6910 33.7424 4.2683
JPEG XR(peg) .95643 33.7518 30.6396 14.1514
JPEG2000 .99981 34.9241 33.804 13.3235
WebP (default) .96539 34.2569 34.1609 9.2412

JPEG1(BASE) 1.23542 32.8054 36.398 8.7611
JPEG XR(peg) 1.22791 35.0437 33.0767 15.6658
JPEG2000 1.24987 36.0575 36.5826 13.3235
WebP (default) 1.21960 35.1127 36.4806 10.6296

JPEG1(BASE) 1.49206 33.7375 38.7158 9.2412
JPEG XR(peg) 1.45315 35.9245 34.7756 17.0048
JPEG2000 1.49907 36.9637 38.294 17.2494
WebP (default) 1.44004 35.7662 38.3353 11.6093

JPEG1(BASE) 1.94791 35.0935 42.047 10.9842
JPEG XR(peg) 1.92128 37.5032 37.8936 19.1876
JPEG2000 1.99992 38.5939 42.6675 18.3036
WebP (default) 1.93790 36.7454 41.645 11.6093

Quality/Compression Comparison for Image /store/pics/core1/cafe.rgb.ppm
Automatically generated by jpegcompare (c) 2005,2007 THOR Software,
Thomas Richter

Codec Size PSNR SSIM PRE VDP PER PQS
mDCT-PSNR VSSIM

JPEG1(BASE) .24440 17.7217 14.9289 -0.0000
JPEG XR(peg) .24568 20.1918 15.745 0.2068
JPEG2000 .24998 20.9265 18.633 -0.0000
WebP (default) .28722 21.0510 19.2176 -0.0000

JPEG1(BASE) .46940 20.9727 21.3728 -0.0000
JPEG XR(peg) .48340 22.8426 20.5372 0.3830
JPEG2000 .49986 23.4697 23.1451 0.0341
WebP (default) .48871 23.0706 23.5431 0.2068

JPEG1(BASE) .73124 22.9484 26.1397 -0.0000
JPEG XR(peg) .74165 24.8673 24.0598 1.1647
JPEG2000 .74997 25.4124 26.2414 0.1720
WebP (default) .73112 24.8308 27.0273 0.2418

JPEG1(BASE) .98710 24.2322 29.2305 0.4187
JPEG XR(peg) .95272 26.1691 26.2559 4.4939
JPEG2000 .99992 26.8626 28.6475 3.2206
WebP (default) .96825 26.0927 29.4211 0.4905

JPEG1(BASE) 1.22464 25.1629 31.4832 0.6358
JPEG XR(peg) 1.24882 27.6840 28.6664 6.4036
JPEG2000 1.24979 28.1100 31.146 4.1029
WebP (default) 1.20185 26.9971 31.2811 0.5993

JPEG1(BASE) 1.47992 25.9890 33.6122 0.6358
JPEG XR(peg) 1.45572 28.6012 30.1243 6.1926
JPEG2000 1.49987 29.2380 32.1785 6.1926
WebP (default) 1.46361 27.8070 33.1717 0.8586

JPEG1(BASE) 1.99688 27.3314 37.1431 0.6726
JPEG XR(peg) 1.98610 30.5077 32.9817 10.7462
JPEG2000 1.99969 30.9888 34.9296 7.4623
WebP (default) 1.95300 28.9174 36.2582 1.8957

Reply all

Reply to author

Forward

0 new messages