Lossless compression of existing JPEG files

230 views
Skip to first unread message

Darryl Lovato

unread,
Jan 7, 2005, 2:36:23 PM1/7/05
to
Yesterday, Allume Systems, a division of IMSI (and creators of the popular
"StuffIt" compression technology) announced a new technology which allows
users and developers to losslessly recompress JPEG files an average of 30%
smaller than the original JPEG file (as well as other Compressed data
types/files), WITHOUT additional data loss.

While the "Compression" of existing compressed files has thus far been
viewed as "impossible", the company has acquired and further developed, and
submitted patents, on a technology which allows for Jpeg to be further
compressed. The method is applicable to other compressed data types (Zip,
MPEG, MP3 and others) to be losslessly re-compressed.

This technology results in a smaller file than the original compressed data
with no data loss.

Working Pre-release test tools have been sent to (and verified by)
independent compression test sites, including:

<http://www.maximumcompression.com/>
<http://www.compression.ca/>

The new technology does NOT break any Information Theory Laws, and will be
shipped later this qtr in commercial products as well as be available for
licensing. The new technology does NOT compress "random files", but rather
previously "compressed files" and "compressed parts" of files. The
technology IS NOT recursive.

The company has filed patents on the new technologies.

The press releases regarding the technology can be found here:

<http://www.allume.com/company/pressroom/releases/stuffit/010605stuffit9.htm
l>
<http://www.allume.com/company/pressroom/releases/stuffit/010605jpeg.html>

Additionally, a white paper has been posted which details the companies
expansion into image compression from it's traditional lossless
archiving/text compression focus, along with results of the technology.

http://www.stuffit.com/imagecompression/

These technologies will be included in future versions of the StuffIt
product line as well as new products and services, and technology licenses
available from Allume and IMSI.

The core technology will also be licensed to companies in the Medical,
Camera, Camera Phone, Image management, internet acceleration, and many
other product areas.

- Darryl

Jeff Gilchrist

unread,
Jan 7, 2005, 4:03:40 PM1/7/05
to
Unlike others claiming to be able to compress already compressed data,
Allume actually sent me working code so I could verify for myself that
what they claim is true.

The test program they sent me is a beta version and not the final
product so things could change for the commercial release. The test
program they sent was a single Windows executable (.exe) that is
118,784 bytes in size and performs both compression and uncompression.
The executable itself is not compressed and UPX would bring it down to
about 53KB.

I will describe the method of testing I used to show that no tricks
were being performed and nothing could "fool" me into thinking the
algorithm was working if it was not. I placed the compressor onto a
floppy disk and copied the .exe file to Machine A and Machine B. Both
machines have no way to communicate with each other and are not
connected to any wired or wireless network. I used my Nikon Coolpix
3MP digital camera to generate JPEG files for use in the test. The
three image files were copied to Machine A. They were not copied to
and did not previously exist on Machine B. On Machine A, the SHA-1
hash of the 3 JPEG files were taken and the digests written down. The
Allume software was used to individually compress the 3 JPEG files and
sure enough they all got around 30% compression. The compressed files
only were copied to a floppy disk and then walked over to Machine B.
They were then copied to Machine B and using the Allume executable
already installed, the 3 JPEG files were uncompressed. The file sizes
on Machine B matched the originals on Machine A which is a good start.
The images were viewable in a JPEG reader as expected. Finally the
SHA-1 hashes of the uncompressed files on Machine B were taken and
confirmed to match the SHA-1 hashes of the original JPEG files on
Machine A which means they are bit for bit identical. This is not a
hoax, but the algorithm actually works.

Here are some details from my testing and how their compression
algorithm compares to some popular ones. Even if you don't want to
believe me, this algorithm should be available in the next release of
their compression product so you can verify it for yourself.


Test JPEGs:

DSCN3974.jpg (National Art Gallery, Ottawa, Canada)
File size : 1114198 bytes
Resolution : 2048 x 1536
Jpeg process: Baseline (Fine Compression)
SHA-1 Hash : f6b3b306f213d3f7568696ed6d94d36e58b4ce1b

DSCN4465.jpg (Golden Pagoda, Kyoto, Japan)
File size : 694895 bytes
Resolution : 2048 x 1536
Jpeg process: Baseline (Normal Compression)
SHA-1 Hash : 5f3d92f558d7cc2d850aa546ae287fa7b61f890d

DSCN5081.jpg (AI Building, MIT, USA)
File size : 516726 bytes
Resolution : 2048 x 1536
Jpeg process: Baseline (Fine Compression)
SHA-1 Hash : 3dcf29223076c4acae5108f6d2fa04cd1ddc5e70

Test Machine: P4 1.8GHz, 512MB RAM, Win2000


Results
=======

DSCN3974.jpg (1,114,198 bytes)

Program Comp Time Uncomp Time Compressed Size % Smaller
------- --------- ----------- --------------- ---------
Allume JPEG 7.9 sec 8.4 sec 835,033 bytes 25.0%
bzip2 1.02 1.6 sec 0.5 sec 1,101,627 bytes 1.1%
7-Zip 3.13 (PPMd) 4.3 sec 3.9 sec 1,102,032 bytes 1.1%
zip 2.3 -9j 0.2 sec 0.1 sec 1,104,866 bytes 0.8%
rar 3.42 -m5 1.7 sec 0.1 sec 1,107,336 bytes 0.6%
7-Zip 3.13 (LZMA) 2.6 sec 0.4 sec 1,113,492 bytes 0.1%

DSCN4465.jpg (694,895 bytes)

Program Comp Time Uncomp Time Compressed Size % Smaller
------- --------- ----------- --------------- ---------
Allume JPEG 5.8 sec 6.1 sec 526,215 bytes 24.3%
bzip2 1.02 1.0 sec 0.3 sec 683,344 bytes 1.7%
zip 2.3 -9j 0.1 sec 0.1 sec 683,462 bytes 1.6%
rar 3.42 -m5 1.2 sec 0.1 sec 685,283 bytes 1.4%
7-Zip 3.13 (PPMd) 2.5 sec 2.4 sec 687,425 bytes 1.1%
7-Zip 3.13 (LZMA) 1.6 sec 0.3 sec 689,264 bytes 0.8%

DSCN5081.jpg (516,726 bytes)

Program Comp Time Uncomp Time Compressed Size % Smaller
------- --------- ----------- --------------- ---------
Allume JPEG 5.8 sec 6.0 sec 374,501 bytes 27.5%
7-Zip 3.13 (PPMd) 2.0 sec 1.8 sec 504,718 bytes 2.3%
rar 3.42 -m5 0.8 sec 0.1 sec 505,296 bytes 2.2%
zip 2.3 -9j 0.1 sec 0.1 sec 505,334 bytes 2.2%
bzip2 1.02 0.7 sec 0.2 sec 506,714 bytes 1.9%
7-Zip 3.13 (LZMA) 1.2 sec 0.2 sec 508,449 bytes 1.6%


A couple of sample JPGs sent by Allume showed even better compression
performance. One JPG (1610 x 3055) with a file size of 315,085 bytes
compressed by 54.8% to 142,281 bytes. A second sample JPG (1863 x
2987) with a file size of 40,367 bytes compressed by 90.9% to 3,656
bytes. Your mileage will vary.
Regards,
Jeff Gilchrist
(www.compression.ca)

Fulcrum

unread,
Jan 7, 2005, 4:50:00 PM1/7/05
to
My testing method was similar to Jeff's but I didn't use a SHA-1 hash
but did a binary diff between the original and the
compressed/decompressed jpeg. The average compression rate I got was a
bit less then 30% (around 24%). Almost all files I tested scored
compression ratio's over 20%.
---
Regards,
Werner Bergmans
(www.maximumcompression.com)

Phil Carmody

unread,
Jan 7, 2005, 8:12:54 PM1/7/05
to
"Jeff Gilchrist" <jsgil...@hotmail.com> writes:
> Program Comp Time Uncomp Time Compressed Size % Smaller
> ------- --------- ----------- --------------- ---------
> Allume JPEG 7.9 sec 8.4 sec 835,033 bytes 25.0%
> bzip2 1.02 1.6 sec 0.5 sec 1,101,627 bytes 1.1%
> 7-Zip 3.13 (PPMd) 4.3 sec 3.9 sec 1,102,032 bytes 1.1%
> zip 2.3 -9j 0.2 sec 0.1 sec 1,104,866 bytes 0.8%
> rar 3.42 -m5 1.7 sec 0.1 sec 1,107,336 bytes 0.6%
> 7-Zip 3.13 (LZMA) 2.6 sec 0.4 sec 1,113,492 bytes 0.1%

The comparison of a specific JPEG compressor (i.e compressor
of JPEGs) against a general purpose compressor does not seem
particularly informative. It's not comparing like with like.
What happens when you run Allume JPEG on the calgary corpus?

It's long been known that JPEG is suboptimal in many places.
However, it does appear that Allume have demonstrated by how
much JPEG can be improved, much more than I expected, and so
all kudos to them for that remarkable feat. Great work guys!

Phil


--
The gun is good. The penis is evil... Go forth and kill.

Phil Carmody

unread,
Jan 7, 2005, 8:15:44 PM1/7/05
to
Darryl Lovato <dlo...@allume.com> writes:
> The new technology does NOT compress "random files", but rather
> previously "compressed files" and "compressed parts" of files.

You're not cranks - I'd change the wording of that sentence a bit,
as it says too much, and could be deliberately misinterpreted such
that it actaully is demonstrably false.

Darryl Lovato

unread,
Jan 7, 2005, 8:24:33 PM1/7/05
to
On 1/7/05 5:12 PM, in article 87u0psa...@nonospaz.fatphil.org, "Phil
Carmody" <thefatphi...@yahoo.co.uk> wrote:

> "Jeff Gilchrist" <jsgil...@hotmail.com> writes:
>> Program Comp Time Uncomp Time Compressed Size % Smaller
>> ------- --------- ----------- --------------- ---------
>> Allume JPEG 7.9 sec 8.4 sec 835,033 bytes 25.0%
>> bzip2 1.02 1.6 sec 0.5 sec 1,101,627 bytes 1.1%
>> 7-Zip 3.13 (PPMd) 4.3 sec 3.9 sec 1,102,032 bytes 1.1%
>> zip 2.3 -9j 0.2 sec 0.1 sec 1,104,866 bytes 0.8%
>> rar 3.42 -m5 1.7 sec 0.1 sec 1,107,336 bytes 0.6%
>> 7-Zip 3.13 (LZMA) 2.6 sec 0.4 sec 1,113,492 bytes 0.1%
>
> The comparison of a specific JPEG compressor (i.e compressor
> of JPEGs) against a general purpose compressor does not seem
> particularly informative. It's not comparing like with like.
> What happens when you run Allume JPEG on the calgary corpus?

Depends on how you look at it - this technology will be shipped in StuffIt
(as well as other things), so you can substitute the "Allume JPEG" title of
the test tool with "StuffIt" - a general purpose compression product, just
like the others.

> It's long been known that JPEG is suboptimal in many places.
> However, it does appear that Allume have demonstrated by how
> much JPEG can be improved, much more than I expected, and so
> all kudos to them for that remarkable feat. Great work guys!

Thank you.

- Darryl

> Phil

Darryl Lovato

unread,
Jan 7, 2005, 8:50:50 PM1/7/05
to
On 1/7/05 5:15 PM, in article 87pt0ga...@nonospaz.fatphil.org, "Phil
Carmody" <thefatphi...@yahoo.co.uk> wrote:

> Darryl Lovato <dlo...@allume.com> writes:
>> The new technology does NOT compress "random files", but rather
>> previously "compressed files" and "compressed parts" of files.
>
> You're not cranks - I'd change the wording of that sentence a bit,
> as it says too much, and could be deliberately misinterpreted such
> that it actaully is demonstrably false.

Phil,

It doesn't say "ALL compressed files" or even "ALL compressed parts of
files", but what is stated above is factual - it does compress previously
compressed files, and/or, previously compressed parts of files - it just
depends on what those "compressed files, and compressed parts of files are".

I DO understand what you are saying, however - and believe me, we want to
distance ourselves from what has gone on here (comp.compression) in the past
regarding hoaxes.

So, for now, I'll refine the statement to compressed jpeg files, and parts
of files that include jpegs (the technology covers more than JPEG, but since
that's all we submitted for independent benchmarking/testing/verification so
far, I'm OK with limiting the statement to that at this time).

Thanks for pointing this out.

- Darryl

> Phil

Phil Carmody

unread,
Jan 7, 2005, 10:50:30 PM1/7/05
to
Darryl Lovato <dlo...@allume.com> writes:
> I DO understand what you are saying, however - and believe me, we want to
> distance ourselves from what has gone on here (comp.compression) in the past
> regarding hoaxes.

Absolutely. That's why that particular sentence seemed to stand out
just a little.


Side note - is there a maintainer for the comp.compression FAQ?
The "some compressions schemes leave room for further compression"
and "recursive compression" concepts probably need to have a great
fat wedge driven between them, lest loons or innocents confuse the
two.


Cheerio,

Alexis Gallet

unread,
Jan 8, 2005, 3:54:47 AM1/8/05
to
Hi,

very interesting results indeed ! I'm curious, I'd like to ask you a few
questions :

* how does the lossless recompression rate varies with the JPEG's quality
factor ? I would expect that the lower the quality factor of the original
JPEG, the better the recompression works...?

* About the JPEG files used for the tests : were they baseline or baseline
optimized JPEGs ? would one get similar recompression ratios with JPEGs
generated by the famous IJG library ?

Anyway, I'm impressed by the figures ! It would be very interesting to run a
JPEG+stuffIt versus JPEG2000 comparison on your test files, eg by plotting
(x=bitrate, y=PSNR) graph...

Regards,
Alexis Gallet


Severian

unread,
Jan 8, 2005, 4:20:59 AM1/8/05
to
On 7 Jan 2005 13:03:40 -0800, "Jeff Gilchrist"
<jsgil...@hotmail.com> wrote:

You don't mention the processor types and speeds involved, but unless
the time required for this extra compression is reduced considerably
in the released product, it will have very limited usage until
processors are considerably faster.

Disk space and bandwidth are both relatively cheap; making users wait
7-10 seconds to save or view an extra-compressed image (vs. my
estimate of 1-3 seconds for the original JPEG) is simply annoying.

>A couple of sample JPGs sent by Allume showed even better compression
>performance. One JPG (1610 x 3055) with a file size of 315,085 bytes
>compressed by 54.8% to 142,281 bytes.

That's likely a low-quality JPEG to begin with. Why care that the
recompression is lossless? The image most likely looks like shit
already.

>A second sample JPG (1863 x
>2987) with a file size of 40,367 bytes compressed by 90.9% to 3,656
>bytes. Your mileage will vary.

At that compression, the original JPEG is useless noise, or at least a
useless image.

I'm not surprised that JPEG-compressed data can be compressed further;
but the time required for the extra compression will make it initially
useful only in fringe applications (archival storage, etc.), until it
is reasonably quick.

By that time, JPEG2000 or newer methods should be ubiquitous, and
provide better results at least as quickly. Also, they handle the
higher color depths necessary for decent digital photography.

--
Sev

Jeff Gilchrist

unread,
Jan 8, 2005, 6:48:35 AM1/8/05
to
Hi Severian,

Actually I do mention processor types and speeds. If you re-read my
post you will find:

"Test Machine: P4 1.8GHz, 512MB RAM, Win2000"

The sample files are special cases. I saw around 25% compression with
my own files. They only claim 30% on average. I was just pointing out
what the algorithm can do. The sample images do not look like "shit"
already, but even if they did not look that great, you would not want
to lose any more quality.

>From what the company has said, they will be including the algorithm in
their Stuffit archiving software so it will be used for what you
suggest (archival storage, etc...).

Regards,
Jeff.

Jeff Gilchrist

unread,
Jan 8, 2005, 6:52:56 AM1/8/05
to
Phil,

Their JPEG algorithm will be part of a general purpose compressor so it
seems like a good comparison to me. Also, many people are not
"experts" in compression so do not even realize that programs such as
ZIP and RAR get almost no compression from image data like JPEG. I was
also posting the details so people could see time-wise how the
algorithm compares in speed.

Regards,
Jeff.

Darryl Lovato

unread,
Jan 8, 2005, 10:58:14 AM1/8/05
to
On 1/8/05 12:54 AM, in article 41df9fda$0$31792$626a...@news.free.fr,
"Alexis Gallet" <alexis.galle...@free.fr> wrote:

> Hi,
>
> very interesting results indeed ! I'm curious, I'd like to ask you a few
> questions :
>
> * how does the lossless recompression rate varies with the JPEG's quality
> factor ? I would expect that the lower the quality factor of the original
> JPEG, the better the recompression works...?

The more initial JPEG image loss, the more %additional lossless compression
with our technology. I.e If someone is really concerned about file size,
they can go as far as possible with JPEG without seeing visual artifacts,
then use our technology on the result to get it even smaller w/no additional
loss.

Here are results for the Kodak image set (24 files) with various JPEG
quality settings:

Uncompressed JPG Level JPG Stuffit/jpg StuffIt/jpg vs JPG
28,325,140 10 385,260 209,233 45.7%
28,325,140 20 599,579 389,672 35.0%
28,325,140 30 778,323 539,459 30.7%
28,325,140 40 926,698 664,426 28.3%
28,325,140 50 1,068,196 783,370 26.7%
28,325,140 60 1,222,806 912,331 25.4%
28,325,140 70 1,461,298 1,111,188 24.0%
28,325,140 80 1,852,334 1,432,695 22.7%
28,325,140 90 2,767,835 2,172,781 21.5%
28,325,140 100 8,005,968 6,462,394 19.3%

> * About the JPEG files used for the tests : were they baseline or baseline
> optimized JPEGs ? would one get similar recompression ratios with JPEGs
> generated by the famous IJG library ?

It works for any jpeg - baseline, grayscale, progressive. I would expect
that we'd get similar ratio's on any valid JPG file no matter what it was
created with, but haven't specifically tested output from the IJG lib.

> Anyway, I'm impressed by the figures !

Thanks.

>It would be very interesting to run a
> JPEG+stuffIt versus JPEG2000 comparison on your test files, eg by plotting
> (x=bitrate, y=PSNR) graph...

You can get an idea of what it would be by taking an existing comparison
test that someone has done, and reduce the bitrate of JPEG by about 20-40%.

It'll be released to the public in a couple months so feel free to run the
comparison - I'd be interested in the results you get.

> Regards,
> Alexis Gallet
>
>

Matt Mahoney

unread,
Jan 8, 2005, 7:39:50 PM1/8/05
to
"Darryl Lovato" <dlo...@allume.com> wrote in message
news:BE0424B5.15860%dlo...@allume.com...

> Yesterday, Allume Systems, a division of IMSI (and creators of the
popular
> "StuffIt" compression technology) announced a new technology which allows
> users and developers to losslessly recompress JPEG files an average of 30%
> smaller than the original JPEG file (as well as other Compressed data
> types/files), WITHOUT additional data loss.
>
> While the "Compression" of existing compressed files has thus far been
> viewed as "impossible", the company has acquired and further developed,
and
> submitted patents, on a technology which allows for Jpeg to be further
> compressed. The method is applicable to other compressed data types (Zip,
> MPEG, MP3 and others) to be losslessly re-compressed.

Interesting. I guess the idea might be to uncompress the data, then
compress it with a better model. This is probably tricker for lossy formats
like JPEG than lossless ones like Zip.

> Working Pre-release test tools have been sent to (and verified by)
> independent compression test sites, including:
>
> <http://www.maximumcompression.com/>

The benchmark has one JPEG file, so far not yet updated. The best
compression currently is 3.5% (WinRK 2.0). (There is also a .bmp file that
was converted from a JPEG so it has JPEG like artifacts that make it easier
to compress losslessly.)

> <http://www.compression.ca/>

This benchmark hasn't been updated since 2002 and doesn't have any JPEG
files. However I'm not aware of any other benchmarks that do.

-- Matt Mahoney


Severian

unread,
Jan 8, 2005, 7:51:39 PM1/8/05
to
On 8 Jan 2005 03:48:35 -0800, "Jeff Gilchrist"
<jsgil...@hotmail.com> wrote:

>Hi Severian,
>
>Actually I do mention processor types and speeds. If you re-read my
>post you will find:
>
>"Test Machine: P4 1.8GHz, 512MB RAM, Win2000"

Sorry, I missed it, even though I looked for it.

>The sample files are special cases. I saw around 25% compression with
>my own files. They only claim 30% on average. I was just pointing out
>what the algorithm can do. The sample images do not look like "shit"
>already, but even if they did not look that great, you would not want
>to lose any more quality.

I haven't seen them, but I find it hard to believe the second example
is useful; perhaps it's an inappropriate file for JPEG compression in
the first place.

As far as losing more quality, that is true.

>>From what the company has said, they will be including the algorithm in
>their Stuffit archiving software so it will be used for what you
>suggest (archival storage, etc...).

Yes, but it also makes their announcement a bit less amazing than they
seem to want everyone to believe!

Anyone seriously dealing with images would archive the originals
(losslessly compressed). I'm not sure I understand the market for this
new compression. Porn collectors? Google.com and archive.org?

It just doesn't seem to be as big a deal as they make it out to be.

--
Sev

Darryl Lovato

unread,
Jan 8, 2005, 8:59:45 PM1/8/05
to
On 1/8/05 4:51 PM, in article dlv0u09m1jv8t7r2f...@4ax.com,
"Severian" <seve...@chlamydia-is-not-a-flower.com> wrote:

> On 8 Jan 2005 03:48:35 -0800, "Jeff Gilchrist"
> <jsgil...@hotmail.com> wrote:
>
>> Hi Severian,
>>
>> Actually I do mention processor types and speeds. If you re-read my
>> post you will find:
>>
>> "Test Machine: P4 1.8GHz, 512MB RAM, Win2000"
>
> Sorry, I missed it, even though I looked for it.
>
>> The sample files are special cases. I saw around 25% compression with
>> my own files. They only claim 30% on average. I was just pointing out
>> what the algorithm can do. The sample images do not look like "shit"
>> already, but even if they did not look that great, you would not want
>> to lose any more quality.
>
> I haven't seen them, but I find it hard to believe the second example
> is useful; perhaps it's an inappropriate file for JPEG compression in
> the first place.

The two sample files I sent Jeff were:

A portrait of a pretty young lady. (fully clothed) :-)
A NASA/Space Image of a star cluster.

The JPEG compression was not high enough to make noticeable artifacts in
either image. As stated elsewhere, the more compression jpeg gets, the
further % reduction we can get, so higher jpeg settings do make a
difference, but the nature of the original pre-jpeg'd image also makes a
difference.

> As far as losing more quality, that is true.
>
>>> From what the company has said, they will be including the algorithm in
>> their Stuffit archiving software so it will be used for what you
>> suggest (archival storage, etc...).
>
> Yes, but it also makes their announcement a bit less amazing than they
> seem to want everyone to believe!

Tons of users send and store jpegs. Being able to compress them, even 20%
(about the least we get), without creating additional image loss adds up.

Many times, the user no longer has the original image (camera's storing
directly into jpeg on the compact flash card, etc). You can always take an
image, and recompress it with a higher jpeg compression setting, but that
makes the original JPEG "loss" permanent, AND adds more loss. Our
technology allows you to reduce the size without messing with the quality at
all.

> Anyone seriously dealing with images would archive the originals
> (losslessly compressed). I'm not sure I understand the market for this
> new compression. Porn collectors? Google.com and archive.org?

I'm sure it will be used for all the above :-)

> It just doesn't seem to be as big a deal as they make it out to be.

I suppose it depends on the user. Going from 1-3% compression of JPEGs to
20-40% (in some fringe cases much more) is a pretty significant advancement
to the field of lossless compression IMHO. Especially given the average
"large size" and "large popularity" of these files.

You have no idea how many people, in the past, put a JPEG (or many jpeg
files) into a StuffIt archive, then complain to us that we barely compressed
it at all. Even "reviewers", that should know better, do this more often
than you would think. The technology we announced will be applied to other
compressed file types as well.

- Darryl

> --
> Sev

code_wrong

unread,
Jan 8, 2005, 9:24:58 PM1/8/05
to

"Darryl Lovato" <dlo...@allume.com> wrote in message
news:BE05CF2A.158FB%dlo...@allume.com...

It brilliant .. when can we see the algorithm ;-)
or the patents even


Darryl Lovato

unread,
Jan 8, 2005, 9:53:25 PM1/8/05
to
On 1/8/05 6:24 PM, in article 110523747...@demeter.uk.clara.net,
"code_wrong" <t...@tac.ouch.co.uk> wrote:

<Snipped out a bunch of stuff not relevant to this reply>

>>> It just doesn't seem to be as big a deal as they make it out to be.
>>
>> I suppose it depends on the user. Going from 1-3% compression of JPEGs to
>> 20-40% (in some fringe cases much more) is a pretty significant
>> advancement
>> to the field of lossless compression IMHO. Especially given the average
>> "large size" and "large popularity" of these files.
>>
>> You have no idea how many people, in the past, put a JPEG (or many jpeg
>> files) into a StuffIt archive, then complain to us that we barely
>> compressed
>> it at all. Even "reviewers", that should know better, do this more often
>> than you would think. The technology we announced will be applied to
>> other
>> compressed file types as well.
>
> It brilliant .. when can we see the algorithm ;-)
> or the patents even

Our patent lawyer has advised us to not give out details of the inventions
at this time. There are actually 2 patents.

I assume you can all see the patents when the patent office posts it to
their site - it describes the processes involved. I'm not really sure when
it will be posted on their site, the full utility patent application was
submitted prior to our announcement (and a provisional application a few
months ago as well).

The first consumer versions of software (StuffIt, etc) that include the new
technologies/inventions will be shipped late this qtr. Licensed versions
will be available shortly thereafter.

You'll have to take our word for it (and more importantly Werner Bergmans,
and Jeff Gilchrists' independent verification) for now.

It works.

- Darryl

Darryl Lovato

unread,
Jan 8, 2005, 11:11:02 PM1/8/05
to
On 1/8/05 4:39 PM, in article
q3%Dd.1638$KJ2...@newsread3.news.atl.earthlink.net, "Matt Mahoney"
<matma...@yahoo.com> wrote:

<snip>

>> Working Pre-release test tools have been sent to (and verified by)
>> independent compression test sites, including:
>>
>> <http://www.maximumcompression.com/>
>
> The benchmark has one JPEG file, so far not yet updated. The best
> compression currently is 3.5% (WinRK 2.0). (There is also a .bmp file that
> was converted from a JPEG so it has JPEG like artifacts that make it easier
> to compress losslessly.)
>
>> <http://www.compression.ca/>
>
> This benchmark hasn't been updated since 2002 and doesn't have any JPEG
> files.

They posted "verification" as a reply directly to the newsgroup as a
follow-up to my original message - I'm not sure when their web sites will be
updated.

> However I'm not aware of any other benchmarks that do.

Agreed, the only test/benchmarking site that includes a JPEG as a test file
for lossless compression (www.maximumcompression). It did so (it appears)
in order to test worst case performance of programs, because this has
previously been viewed as "impossible" - nobody has pulled it off.

Just an understandable, but unfortunate, association of lossless compression
of "compressed data" and lossless compression of "random data" together.

Part of my job is to look into "compression claims" - no matter how crazy
they might seem (I've talked to a lot of people in the past because of this
- I won't mention names, but we all know who they are) - on the odd chance
that someone did have something.

It's my responsibility to make sure we don't "miss something" important.
Anyway, I understand the questions, and pessimism, which is why we felt it
was important to send working test tools to Jeff and Werner for independent
verification. Having an open mind is a good thing - without it, I would
have passed on this invention - that turned out, works! :-)

- Darryl

> -- Matt Mahoney
>
>

Uwe Herklotz

unread,
Jan 9, 2005, 3:52:00 AM1/9/05
to
"Darryl Lovato" <dlo...@allume.com> wrote:

> The more initial JPEG image loss, the more %additional lossless
compression
> with our technology. I.e If someone is really concerned about file size,
> they can go as far as possible with JPEG without seeing visual artifacts,
> then use our technology on the result to get it even smaller w/no
additional
> loss.

Very interesting results!
Does your new technology work also for JPEGs created in lossless mode?
If yes, what %additional lossless compression is possible in this case?
What happens if random data is enclosed in a lossless JPEG picture and
this file is then re-compressed with your technology? Surely you will
not be able to get a result smaller than size of original random data.

Regards
Uwe


Uwe Herklotz

unread,
Jan 9, 2005, 3:52:20 AM1/9/05
to
"Matt Mahoney" <matma...@yahoo.com> wrote:

> Interesting. I guess the idea might be to uncompress the data, then
> compress it with a better model. This is probably tricker for lossy
formats
> like JPEG than lossless ones like Zip.

I also thought about such an idea. But even for Zip files it seems
to be very difficult. Uncompressing the data and recompressing with
a better model is easy. But how to ensure that this can be reversed
i.e. it must be possible to recover the original Zip file. Without
knowing the parameters of the original Zip compression this is very
difficult, maybe impossible at all.

Regards
Uwe


Darryl Lovato

unread,
Jan 9, 2005, 5:10:01 AM1/9/05
to
On 1/9/05 12:52 AM, in article 34c9m9F...@individual.net, "Uwe
Herklotz" <_no_s...@yahoo.com> wrote:

> "Darryl Lovato" <dlo...@allume.com> wrote:
>
>> The more initial JPEG image loss, the more %additional lossless
> compression
>> with our technology. I.e If someone is really concerned about file size,
>> they can go as far as possible with JPEG without seeing visual artifacts,
>> then use our technology on the result to get it even smaller w/no
> additional
>> loss.
>
> Very interesting results!
> Does your new technology work also for JPEGs created in lossless mode?
> If yes, what %additional lossless compression is possible in this case?

The tool I created the JPEG's with (previous posted test results show)
allows quality of 100, which is the near lossless mode - and we still get
about 20% additional compression. Again, this technology is applicable to
many other compressed types other than regular lossy JPEG files. Initially
we picked JPEG as the first compressed file type to ship support for -
mostly due to the mass number of JPEG files out there - but we will roll out
support for other file types over the coming year as we have time to tune
and test the technology with other compressed file types.

> What happens if random data is enclosed in a lossless JPEG picture and
> this file is then re-compressed with your technology? Surely you will
> not be able to get a result smaller than size of original random data.

As I've stated before, we don't claim to compress random data only some
types of compressed data.

I haven't specifically tried to compress a JPEG of a picture that is
essentially "random" pixels to start with, but I doubt we would be able to
do as well - if anything in that case. This is an educated guess based on
the fact that the more compressible the original picture is via JPEG, the
more we get, so the opposite should also be true. This really doesn't
matter that much though in practice, since I don't expect many users have
jpeg files that contain pixels of random noise :-)

If you have such a JPEG, send it to me and I'll tell you what happens.

- Darryl

> Regards
> Uwe
>
>

Alexis Gallet

unread,
Jan 9, 2005, 6:10:42 AM1/9/05
to
"Uwe Herklotz" <_no_s...@yahoo.com> wrote:

> I also thought about such an idea. But even for Zip files it seems
> to be very difficult. Uncompressing the data and recompressing with
> a better model is easy. But how to ensure that this can be reversed
> i.e. it must be possible to recover the original Zip file. Without
> knowing the parameters of the original Zip compression this is very
> difficult, maybe impossible at all.

Indeed ! Even with given parameters (eg size of the sliding window), I think
there are (in general) several valid zip files that correspond to the same
original file.

On the other hand, the JPEG case looks much easier to me, because I don't
think there is this uniqueness issue in the JPEG format.

On the compression side, you would have to :
1) entropy decode the JPEG file (ie, Huffman decode all the coeffs, then
DPCM decode the DC coeffs and RLE decode the AC coeffs) - but don't
unquantize the coeffs neither perform an inverse DCT
2) reencode the coeffs using a context-adaptive arithmetic coder, with
contexts tuned for 8x8 DCT data. Prior to that, one could further
decorrelate the DC coeffs by applying once or twice a reversible wavelet
filter (eg the 5/3 integer filter) on them. But the major compression gain
would imo be obtained by using the correlation between the AC coeffs of
adjacent blocks (same frequency & adjacent spatial locations), which is
completely ignored in JPEG.
3) don't forget to copy in the archive the huffman tables from the header of
the original JPEG file. Although they aren't needed to recover losslessly
the image itself, they are needed to recover a JPEG file which is
bit-identical to the original JPEG.

On the decompression side :
1) entropy decode the archive
2) reencode it with DPCM/RLE followed by Huffman, using the original JPEG
file's tables

Still, I'm not sure this scheme would yield a 20% improvement on the
high-quality JPEGs (quality factor >= 80)... I think that's the part where
Allume's results are the most impressive. And the slowness of their
algorithm (according to Jeff Gilchrist's figures) suggests that they are
doing something more sophisticated than what I'm guessing... Anyone got an
idea ?

Regards,
Alexis Gallet

Phil Carmody

unread,
Jan 9, 2005, 8:51:03 AM1/9/05
to
Severian <seve...@chlamydia-is-not-a-flower.com> writes:
> Anyone seriously dealing with images would archive the originals
> (losslessly compressed). I'm not sure I understand the market for this
> new compression. Porn collectors? Google.com and archive.org?
>
> It just doesn't seem to be as big a deal as they make it out to be.

I think it's more a geek thing. One-upmanship.

I remember people who used LHA, or ARJ or LZH or whatever it was that
was better than PKZip back in the late 80s or early 90s, waltzing around
the place pretending to be oh-so-superior to the lamo's that used the
utterly mediocre PKZip.

Never underestimate the gadget-affinity of geeks.

Anton Kratz

unread,
Jan 9, 2005, 9:01:43 AM1/9/05
to

"Jeff Gilchrist" <jsgil...@hotmail.com> wrote:

> Unlike others claiming to be able to compress already compressed data [...]

Why do so many people think it is not possible to compress
already compressed data? It's easy: take alice29.txt for example
from the canterbury corpus, compress it with a RLE compressor.
result: a slightly compressed file because of very few long runs of spaces
in the formatting. The file is compressed, okay? Now, compress it with
Huffman or whatever. Now you see it is more compressed.

It is not possible to compress already compressed data with the *same*
compression algo you used in the first pass. Maybe that is what you mean?!

Anton


Fulcrum

unread,
Jan 9, 2005, 10:22:24 AM1/9/05
to
> > Working Pre-release test tools have been sent to (and verified by)
> > independent compression test sites, including:
> >
> > <http://www.maximumcompression.com/>
>
> The benchmark has one JPEG file, so far not yet updated. The best
> compression currently is 3.5% (WinRK 2.0). (There is also a .bmp
file that
> was converted from a JPEG so it has JPEG like artifacts that make it
easier
> to compress losslessly.)
>
> -- Matt Mahoney

I will update my website when the new Stuffit with this technology is
shipped. The application I tested is just an experimental testbed to
compress jpeg's.

Results for my a10.jpg file:
On my AMD Athlon 1800+ a10.jpg 842468 -> 643403 76.37% in about 4 sec
---
Regards,
Werner Bergmans

Fulcrum

unread,
Jan 9, 2005, 10:38:44 AM1/9/05
to
> Agreed, the only test/benchmarking site that includes a JPEG as
> a test file for lossless compression (www.maximumcompression).
> It did so (it appears) in order to test worst case performance
> of programs, because this has previously been viewed as
> "impossible" - nobody has pulled it off.
> - Darryl

No, I didn't add it to only test worst case behaviour (I would have
tested the 1 million random digit file instead). But I agree it's a
nice site effect to see some compressors expand the file by 23% after
compressing :)

A long time before I started my site I already noticed some simple
compressors like szip and arj where able to (slightly) compress
jpg-files, and other more advanced compressors like ACE had much more
difficulty to do so. That made it an interessting testcase to add to
the site.


Regards,
Werner Bergmans

code_wrong

unread,
Jan 9, 2005, 11:21:56 AM1/9/05
to

"Phil Carmody" <thefatphi...@yahoo.co.uk> wrote in message
news:878y729...@nonospaz.fatphil.org...

eh? What are you talking about?
This is a million dollar breakthrough


Matt Mahoney

unread,
Jan 9, 2005, 1:01:35 PM1/9/05
to
"Uwe Herklotz" <_no_s...@yahoo.com> wrote in message
news:34c9mhF...@individual.net...

I agree it would be difficult. Perhaps from the zip file you could tell
which program and version and options produced it, then regenerate the zip
file using the exact same algorithm. In theory you could do this by
unzipping and rezipping for each case until you find a match.

The real problem occurs when you come across an unknown format. Given a
string like

..ABC...ABC...ABC

LZ77 allows the third ABC to be coded as a pointer to the first or second
copy, or as literals, or as a mix of literals and pointers. All of these
would unzip correctly. The general solution would be to record which choice
was made, but this would negate most of the compression savings.

I recall a paper which proposed steganographic zip files using the choice of
coding to hide information. I doubt that such files could be compressed
further using this method.

-- Matt Mahoney


Konsta Karsisto

unread,
Jan 9, 2005, 6:59:20 PM1/9/05
to
Matt Mahoney wrote:
> ..ABC...ABC...ABC
>
> LZ77 allows the third ABC to be coded as a pointer to the first or second
> copy, or as literals, or as a mix of literals and pointers. ...

> I recall a paper which proposed steganographic zip files using the choice of
> coding to hide information.

There was a paper in DCC in 2003 or earlier where they used
this redundancy for, I think, error correction. Or, it could
have been something else, too. ;-) Unfortunately, I couldn't
find the reference.


--
KKK

Malcolm Taylor

unread,
Jan 10, 2005, 6:59:22 AM1/10/05
to
Hi Darryl,

Firstly, congrats! It seems that you guys have created something rather
unique. There are many of us who have thought of such things before, but
none of us has managed to make the ideas actually work... :). My own
attempts in WinRK do not come close to the 20+% you reportedly achieve.

Anyway, I do have a question for you. How well does your technology cope
with corrupt JPEG files. For example, if I were to take a JPEG and blat
a few random numbers into it with a hex editor, would your tech still
compress it well?

Malcolm

Phil Carmody

unread,
Jan 10, 2005, 8:16:33 AM1/10/05
to
"code_wrong" <t...@tac.ouch.co.uk> writes:

Yes, you're right, no gadget has ever made a company anything like
a million dollars. Thank you so much for pointing out this undisputable
fact to me and the rest of comp.compression.

Phil Carmody

unread,
Jan 10, 2005, 8:31:38 AM1/10/05
to

Even that's not true. Gzip can repeatedly positively compress at least 3 times.

I would guess that David Scott's BICOM (or at least the algorithm he described
here a few days back) might be able to positively compress 4 or 5 times.
(assuming all zeroes becomes all zeroes.)

I've designed on paper an algorithm which I believe should positively compress
its own output many dozen times if seeded with something like plain text
or html. I don't expect its steady state to be any better than Huffman, though,
so it's only been done as a flight of fancy into _deliberately_ countering
the inaccurate statement that you and others propagate.

All that can be said is:
It's not possible to compress data without redundancies.
Compression programs are designed to remove, to various extents, redundancies.

Design an algorithm to remove the smallest possible amount of redundancy,
and you can be iterating a very large number of times.

Darryl Lovato

unread,
Jan 10, 2005, 10:32:52 AM1/10/05
to
On 1/10/05 3:59 AM, in article 41e26d5e$1...@127.0.0.1, "Malcolm Taylor"
<m...@me.com> wrote:

> Hi Darryl,
>
> Firstly, congrats! It seems that you guys have created something rather
> unique. There are many of us who have thought of such things before, but
> none of us has managed to make the ideas actually work... :). My own
> attempts in WinRK do not come close to the 20+% you reportedly achieve.

Thanks. These technologies are definitely not "simple" to develop, even
after you know they are possible.

> Anyway, I do have a question for you. How well does your technology cope
> with corrupt JPEG files. For example, if I were to take a JPEG and blat
> a few random numbers into it with a hex editor, would your tech still
> compress it well?

I haven't tried it.

- darryl

> Malcolm

Darryl Lovato

unread,
Jan 10, 2005, 11:02:10 AM1/10/05
to
On 1/10/05 5:16 AM, in article 87hdlp5...@nonospaz.fatphil.org, "Phil
Carmody" <thefatphi...@yahoo.co.uk> wrote:

I personally don't mind if what we invented and filed patents for, is called
a "gadget". I'm sure there were people that called the first light bulb a
gadget, the first jet engine a gadget, the first telephone a gadget, the
first personal computer, etc.

We have something that compresses possibly "billions" of files that were
previously "uncompressible" (1-3% isn't significant compression).

These files in question are generally "larger" than the average file size,
they are commonly stored on hard drives, flash cards, backed up, sent via
e-mail, and downloaded/viewed from the web. Now they can be compressed
20-40% on average, and in some cases up to 90%.

This is a very useful, and valuable, "gadget". :-)

And yes, a driving force is "one-upmanship" it's called commercial
competition. There is nothing wrong with trying to make your product better
than the competition. It benefits users, and has happened from the
beginning of time - since "Commerce" was first invented.

- Darryl

> Phil

Anton Kratz

unread,
Jan 10, 2005, 11:43:28 AM1/10/05
to

Hi Phil,

I thought about what you wrote and you are right.

I was wrong indeed when I wrote that you can't compress
already compressed data with the same algo again, because
indeed you can.

But it's still wrong when someone writes that you can't
compress already compressed data (neither with the same algo nor
another one), because in fact you can.


Anton


"Phil Carmody" <thefatphi...@yahoo.co.uk> schrieb im Newsbeitrag news:87d5wd5...@nonospaz.fatphil.org...

Phil Carmody

unread,
Jan 10, 2005, 8:12:23 PM1/10/05
to
Darryl Lovato <dlo...@allume.com> writes:
> On 1/10/05 5:16 AM, in article 87hdlp5...@nonospaz.fatphil.org, "Phil
> Carmody" <thefatphi...@yahoo.co.uk> wrote:

My message id should be in your references header, there's no need to have it
in the body text too.

> > "code_wrong" <t...@tac.ouch.co.uk> writes:
> >> "Phil Carmody" <thefatphi...@yahoo.co.uk> wrote in message
> >> news:878y729...@nonospaz.fatphil.org...

> >>> Never underestimate the gadget-affinity of geeks.
> >>
> >> eh? What are you talking about?
> >> This is a million dollar breakthrough
> >
> > Yes, you're right, no gadget has ever made a company anything like
> > a million dollars. Thank you so much for pointing out this undisputable
> > fact to me and the rest of comp.compression.

I forgot possibly the biggest gadget fad of them all in recent years -
cameras on mobile phones.

> I personally don't mind if what we invented and filed patents for, is called
> a "gadget". I'm sure there were people that called the first light bulb a
> gadget, the first jet engine a gadget, the first telephone a gadget, the
> first personal computer, etc.

Indeed.



> This is a very useful, and valuable, "gadget". :-)
>
> And yes, a driving force is "one-upmanship" it's called commercial
> competition. There is nothing wrong with trying to make your product better
> than the competition. It benefits users, and has happened from the
> beginning of time - since "Commerce" was first invented.

Absolutely. Geek pockets can be fairly deep, and there are certainly large
numbers of them. However, they can be exceptionally fickle too, so even
the best gadget can fail in the market. That's what VC was invented for.

news...@comcast.net

unread,
Jan 12, 2005, 10:25:21 AM1/12/05
to
Phil Carmody <thefatphi...@yahoo.co.uk> wrote:
> Darryl Lovato <dlo...@allume.com> writes:
>> On 1/10/05 5:16 AM, in article 87hdlp5...@nonospaz.fatphil.org, "Phil
>> Carmody" <thefatphi...@yahoo.co.uk> wrote:
>
> My message id should be in your references header, there's no need to have it
> in the body text too.
>
>> > "code_wrong" <t...@tac.ouch.co.uk> writes:
>> >> "Phil Carmody" <thefatphi...@yahoo.co.uk> wrote in message
>> >> news:878y729...@nonospaz.fatphil.org...
>> >>> Never underestimate the gadget-affinity of geeks.
>> >>
>> >> eh? What are you talking about?
>> >> This is a million dollar breakthrough
>> >
>> > Yes, you're right, no gadget has ever made a company anything like
>> > a million dollars. Thank you so much for pointing out this undisputable
>> > fact to me and the rest of comp.compression.
>
> I forgot possibly the biggest gadget fad of them all in recent years -
> cameras on mobile phones.

Unfortunately, taking over 5 seconds on a desktop processor puts it
out of the reasonable range for embedded processors like in phones.

If they can get the time down by an order of magnitude, then things
might start getting more interesting. Under half a second on a
desktop system would mean that you could browse compressed images on a
desktop system fairly painlessly (imagine -- burning 30% more images
on a CD -- great for people who take a lot of digital pictures). But
I'm not going to deal with a lag in image rendering just to fit a
little more on a CD. If you could get it under a second or two on an
embedded processor, then maybe it could be used to increase capacity
directly in things like cameras and phones -- but processor-intensive
algorithms take too long and burn too much power/battery on embedded
devices.

Anyway, I think this is a great development. It's just that they have
too much hype in their original press release (impossible to compress
already compressed data? Pshaw), and they need to improve efficiency
a bit....

--

That's News To Me!
news...@comcast.net

Aleks Jakulin

unread,
Jan 12, 2005, 11:42:39 AM1/12/05
to
Darryl Lovato:
> It works for any jpeg - baseline, grayscale, progressive. I would
> that we'd get similar ratio's on any valid JPG file no matter what
> created with, but haven't specifically tested output from the IJG

Let's rehash some history. JPEG does have the arithmetic coding mode,
which is better than Huffman+RLE combination. However, arithmetic
coding is not used because it was encumbered by patents. Now you're
proposing another solution which is encumbered by patents.

What kind of improvements are you getting as compared with the
arithmetic coding mode of JPEG?

--
mag. Aleks Jakulin
http://www.ailab.si/aleks/
Artificial Intelligence Laboratory,
Faculty of Computer and Information Science,
University of Ljubljana, Slovenia.


John Reiser

unread,
Jan 12, 2005, 1:05:10 PM1/12/05
to
Matt Mahoney wrote:
> [snip] Given a string like

>
> ..ABC...ABC...ABC
>
> LZ77 allows the third ABC to be coded as a pointer to the first or second
> copy, or as literals, or as a mix of literals and pointers. All of these
> would unzip correctly. The general solution would be to record which choice
> was made, but this would negate most of the compression savings.
>
> I recall a paper which proposed steganographic zip files using the choice of
> coding to hide information. [snip]

Please provide more info about that paper.

For a fixed encoder, even if you require that the coded output for the whole
string be the shortest possible, then there still are choices. The choices
form a lattice, the paths through the lattice can be enumerated, and choosing
a specific path conveys log2(total_paths) additional bits of information.
In practice the amount can be 0.1% to a few percent. But it's hardly hidden,
because there are a few obvious canonical paths: always choose the smallest
offset, favor a literal over a match of the same cost (or vice versa), etc.

--

matma...@yahoo.com

unread,
Jan 12, 2005, 4:19:26 PM1/12/05
to

John Reiser wrote:
> Matt Mahoney wrote:
> > [snip] Given a string like
> >
> > ..ABC...ABC...ABC
> >
> > LZ77 allows the third ABC to be coded as a pointer to the first or
second
> > copy, or as literals, or as a mix of literals and pointers. All of
these
> > would unzip correctly. The general solution would be to record
which choice
> > was made, but this would negate most of the compression savings.
> >
> > I recall a paper which proposed steganographic zip files using the
choice of
> > coding to hide information. [snip]
>
> Please provide more info about that paper.

Here is a program that does it. There is some cost in compression.
http://www.mirrors.wiretapped.net/security/steganography/gzip-steg/gzip-steg-README.txt

Guido Vollbeding

unread,
Jan 13, 2005, 3:40:07 AM1/13/05
to
Aleks Jakulin wrote:
>
> Let's rehash some history. JPEG does have the arithmetic coding mode,
> which is better than Huffman+RLE combination. However, arithmetic
> coding is not used because it was encumbered by patents. Now you're
> proposing another solution which is encumbered by patents.

Well, but at least the JPEG arithmetic coding algorithm is published
in detail in the JPEG standard and JPEG book, so anybody who wishes
can reproduce it. And I have done an open and portable implementation
for use with the IJG software for evaluation purposes.

But this "offer" we see here is a pure commercial offer, and no
further information is given about how the algorithm works.
It is a shame that this newsgroup, which was an open-minded
years ago, is now more and more encumbered by commercial interests,
and that many people don't mind and only few people mind.
We see similar behaviour from the commercial JPEG-2000 proponents
here (and it's a fact for me that JPEG-2000 is technically inferior
and thus obsolete, contrary to the false propaganda of its proponents).

I must say that I absolutely don't care about "black-box offers"
which don't provide information about how the algorithm works,
and I think that this is the WRONG newsgroup for such offers
(he should go to some advertising group).

> What kind of improvements are you getting as compared with the
> arithmetic coding mode of JPEG?

At least the JPEG arithmetic coding mode can be used freely in a
few years when the patents expire.

Regards
Guido

Thomas Richter

unread,
Jan 13, 2005, 4:27:40 AM1/13/05
to
Once again, Guido,

> But this "offer" we see here is a pure commercial offer, and no
> further information is given about how the algorithm works.
> It is a shame that this newsgroup, which was an open-minded
> years ago, is now more and more encumbered by commercial interests,
> and that many people don't mind and only few people mind.

I hope you include yourself here in those that "don't mind".

> We see similar behaviour from the commercial JPEG-2000 proponents
> here (and it's a fact for me that JPEG-2000 is technically inferior
> and thus obsolete, contrary to the false propaganda of its proponents).

We observe similar behavour of Guido here who has still failed to
backup his claim by providing published measurements of his
claim. (-; BTW, I'm working for the TU Berlin. I don't need to sell
anything, I'm getting paid either way.

> I must say that I absolutely don't care about "black-box offers"
> which don't provide information about how the algorithm works,
> and I think that this is the WRONG newsgroup for such offers
> (he should go to some advertising group).

I absolutely don't care about people that don't follow scientific
standards. This includes the proposed JPEG compressor unless I'm able
to verify its claims - but which don't look too far off to be correct
- and it also includes your claims about "propaganda" you're so good
in generating yourself. I agree that possibly similar improvements
(at least, in the same range) can be obtained from classical JPEG,
but I'm not the one who bans their compression codec in first place
for that. All I'm saying that I'm not especially excited about it
(thus, no post from my side up to now).

Propaganda: The kind of news that is spread without giving data
to backup its claim.

Interesting that especially you are using this word.

BTW, my data is (still) here:

http://www.math.tu-berlin.de/~thor/imco/Downloads/jpg/

PSNR measurements are here:

http://www.math.tu-berlin.de/~thor/imco/Downloads/jpg/psnr.txt

These are updated measurements with a "arithmetic encoding enabled"
JPEG, waiting there to be verified by other parties (like you) for
quite a while. Source and compressed images are also available at this
URL. Unlike your claim, we have PSNR(j2k) >= PSNR(jpg + arithcoder).
Images with various compression methods and options are found on this
side as well because PSNR is not a very good measurement. The test
image is a pretty "tough" one (lots of edges), I don't need to cheat.
It is pretty large - this is what I always claimed to be necessary.

Now where are your measurements, Guido? Or was this just propaganda?
I'm curious. Still haven't done your homework? Oh well, it's so much
easier to insult people...

Greetings,
Thomas

P.S.: And note that I'm not the one who claims that "JPEG2000" is
always the better choice. It is not. But the word "inferiour" is also
very wrong. It has other applications. (Large images, low-contrast
images, all posted here.)

Guido Vollbeding

unread,
Jan 13, 2005, 5:44:14 AM1/13/05
to
Thomas Richter wrote:
>
> > But this "offer" we see here is a pure commercial offer, and no
> > further information is given about how the algorithm works.
> > It is a shame that this newsgroup, which was an open-minded
> > years ago, is now more and more encumbered by commercial interests,
> > and that many people don't mind and only few people mind.
>
> I hope you include yourself here in those that "don't mind".

I don't like the commercial direction in this newsgroup,
and I myself have no commercial interests or products here.

> We observe similar behavour of Guido here who has still failed to
> backup his claim by providing published measurements of his
> claim. (-; BTW, I'm working for the TU Berlin. I don't need to sell
> anything, I'm getting paid either way.

I have provided working code, and anybody who wishes can perform
their own evaluations. I'm not here to convince other people -
I have done my own tests and drawn my own conclusions.
You have at least done and mentioned a commercial implementation
of your own here.

> BTW, my data is (still) here:
>
> http://www.math.tu-berlin.de/~thor/imco/Downloads/jpg/
>
> PSNR measurements are here:
>
> http://www.math.tu-berlin.de/~thor/imco/Downloads/jpg/psnr.txt
>
> These are updated measurements with a "arithmetic encoding enabled"
> JPEG, waiting there to be verified by other parties (like you) for
> quite a while. Source and compressed images are also available at this
> URL. Unlike your claim, we have PSNR(j2k) >= PSNR(jpg + arithcoder).
> Images with various compression methods and options are found on this
> side as well because PSNR is not a very good measurement. The test
> image is a pretty "tough" one (lots of edges), I don't need to cheat.
> It is pretty large - this is what I always claimed to be necessary.
>
> Now where are your measurements, Guido? Or was this just propaganda?
> I'm curious. Still haven't done your homework? Oh well, it's so much
> easier to insult people...

The sample image you used apparently originates from a digital camera
with mosaic sensor.
I have already pointed out to you earlier that you CAN'T use images
from mosaic sensor cameras for such evaluation!
Mosaic cameras produce *artificial*, unnatural images (they miss 2/3s
of image information which must be artificially calculated afterwards),
and thus can't be used for evaluation with compression technologies
which were developed for *natural* images!
The blurry image results of mosaic cameras benefit the blurry properties
of Wavelet compression. They are both artificial.

Furthermore, you check only low quality domain (level 75 and lower).
The default quantization parameters in standard JPEG were NOT
optimized for low quality domain, therefore your results are
onesided and, again, in favor for your proposed method, which
was deliberately optimized for low quality.
(JPEG quality level 75 or lower is *much* less than what quality-oriented
people in the digital photography domain normally use in practice!)

You see from my critics that all kind of specific tests are
questionable and only good for propaganda purposes.
That's why I urge people to perform their OWN tests with their
OWN material in their OWN configurations with their OWN criteria
and draw their OWN conclusions. Anything else is propaganda.

Regards
Guido

Thomas Richter

unread,
Jan 13, 2005, 8:46:28 AM1/13/05
to
Now, Guido, once again,

> I have provided working code, and anybody who wishes can perform
> their own evaluations.

The data on the mentioned page is generated with that code. Now what?

> I'm not here to convince other people -
> I have done my own tests and drawn my own conclusions.
> You have at least done and mentioned a commercial implementation
> of your own here.

Correct. A tiny nice side income; I'm neither selling the code, nor
does my income depend on the sales or whatever. I've a full position
at the TU that pays my bills if that is what you mean. I'm not getting
a cent more if more licences are sold.

> The sample image you used apparently originates from a digital camera
> with mosaic sensor.
> I have already pointed out to you earlier that you CAN'T use images
> from mosaic sensor cameras for such evaluation!

Well, then please provide some images of your own. Minimum size
1280x1024, BW or color, whatever. Or measure yourself if you don't
thrust me. "lena" is a bit too small for my purpose. (-;

On the other hand, if we assume for a moment that these cameras are
used, and if we assume further that people use them for making images,
and further want to compress images, and further that these images
contain some redundancy due to the process they have been created, how
come that JPEG2000 is able to detect and make use of this redundancy,
but JPEG isn't? How come that if this hardware is even popular that
JPG misses this target? (-;

You see, I can also turn this argument around. An aparent image redunancy
in a popular technology is not made use of. (-; So, even then there
is aparently an interesting market? (-;

> Mosaic cameras produce *artificial*, unnatural images (they miss 2/3s
> of image information which must be artificially calculated afterwards),
> and thus can't be used for evaluation with compression technologies
> which were developed for *natural* images!

Send me an image that you consider natural. Or even better, test yourself,
no problem. The mentioned image is actually pretty tough to compress which
is why I picked it - one of my "stress tests".

> The blurry image results of mosaic cameras benefit the blurry properties
> of Wavelet compression. They are both artificial.

Most blur is rather a result of the limited quality of the optics you
find nowadays in the "claimed to be 4MPixel sector", but then...

> Furthermore, you check only low quality domain (level 75 and lower).

This is one of the regions where JPEG2000 becomes interesting. I've
always said so: i) Large images ii) high compressing iv) low contrast.

The image is not specifically tuned for anything, I haven't
picked it on purpose, but rather to have something with straight high-
contrast edges that are usually a problem for wavelets.

Anyhow, provide your own if you like. Just make sure it they're large
enough. A 800x600 image won't compress very well - I'm not denying it.
The larger the image gets, the more JPEG2000 outperforms it the
traditional technology. And the reason is of course that correlation
lengths in larger images grow over the maximum decorrelation length
of the 8x8 blocks of the JPEG.

> The default quantization parameters in standard JPEG were NOT
> optimized for low quality domain, therefore your results are
> onesided and, again, in favor for your proposed method, which
> was deliberately optimized for low quality.

"Optimized" is not quite the right word, it just works well in this
domain, though not on purpose. Anyhow, feel free to optimize your
quantizer for low compression. If I'm allowed to make a prediction:
You're going to face the same problems: The DCT won't be able to catch
all correlations.

> (JPEG quality level 75 or lower is *much* less than what quality-oriented
> people in the digital photography domain normally use in practice!)

> You see from my critics that all kind of specific tests are