Is PuCrunch the final word in C64 compression (packer/cruncher)?
Aaron Bilger
Back in the day, I was impressed with packers+crunchers I used such as
1001 Crew and TimeCrunch. I didn't understand them at the time, but
thought what they did was intriguing especially for the low memory.
I didn't learn of Huffman coding, Lempel-Ziv, arithmetic coding, etc.
til college years. I still never reverse-engineered a
packer/cruncher, but came to wonder if the packer/cruncher creators
really understood and aimed to implement an algorithm, or were
trial-and-error competing their way to reinventing RLE and sequence
coding.
Searching now, I found excellent info to bring me somewhat up to speed
in C= Hacking issues 16+17 and an info page by Pasi Ojala.
http://www.ffd2.com/fridge/chacking/c=hacking16.txt
http://www.ffd2.com/fridge/chacking/c=hacking17.txt
http://www.cs.tut.fi/~albert/Dev/pucrunch/
Pasi's own approach is very purposeful and thorough, and taking
advantage of a PC for the compression also seems an obvious choice in
the modern world.
My curiosity now is just: was his creation the be-all-end-all of C64
(in-place) compression, or was there any continuing arms race and
improvement after PuCrunch?
Aaron
White Flame (aka David Holz)
news:9c09d1ltq5cvpsc79...@4ax.com...
> My curiosity now is just: was his creation the be-all-end-all of C64
> (in-place) compression, or was there any continuing arms race and
> improvement after PuCrunch?
I believe Exomizer crunches a bit better than PuCrunch, and has gained a lot
of acceptance. The format and compression scheme isn't openly described
like PuCrunch, but all of the code is there.
home page: http://w1.131.comhem.se/~u13114991/exo/ (down?)
also can get it here, under Tools: http://cadaver.homeftp.net/
--
White Flame (aka David Holz)
http://www.white-flame.com/
(spamblock in effect)
wHOAHdUDE
6502Tass cross-compiler v1.3 (and higher) written by Marek Matula.
It realy amazed me. It crunches apps very very small and decrunches very
very fast also. One great piece of work from Marek!
Check it out at the following url:
http://taboo.eu.org/download/index.html
wHOAHdUDE
"Aaron Bilger" <syse...@comcast.net> wrote in message
news:9c09d1ltq5cvpsc79...@4ax.com...
>
Stephan Schmid
> As already indicated, Exomizer has beaten pucrunch on compression ratio
> already a couple of years ago.
>
> The difference in compression ratio is not dramatical, and because
> pucrunch can be used for quite many systems, it is still useful.
It's just in the 2-5% area, IIRC. To me this isn't enough to justify the
long crunch and decrunch times. Especially when batch-crunching 30.001
SIDs that were before turned into C64 runnable programs with PSID64 this
_does_ make a difference (of a few hours). ;-)
Regards,
Stephan
Ojala Pasi 'Albert'
Aaron Bilger <syse...@comcast.net> wrote:
>My curiosity now is just: was his creation the be-all-end-all of C64
>(in-place) compression, or was there any continuing arms race and
>improvement after PuCrunch?
As already indicated, Exomizer has beaten pucrunch on compression ratio
already a couple of years ago.
The difference in compression ratio is not dramatical, and because
pucrunch can be used for quite many systems, it is still useful.
-Pasi
--
"Are you well?"
"I was thinking. Sometimes I think too much."
-- Sulin and Rand in The Wheel of Time:"A Crown of Swords"
iAN CooG
> It's just in the 2-5% area, IIRC. To me this isn't enough to justify
> the long crunch and decrunch times. Especially when batch-crunching
Pucrunch is quite immediate, Exomizer takes more time to pack, but gives
satifactions :).
By the way Exomizer is faster than pucrunch on depack. Used them both
many times.
I've made a little test to show some numbers, sizes are in bytes and
actual blocks occupation.
All the 3 well known crosscrunchers were used:
- Exomizer 2.0beta2 (not yet publicly available)
- Pucrunch 1.11($VER: pucrunch 1.11 30-Jan-2000 W9x fix by iAN CooG)
- Crush 1.1
test on Polygonamy by Steve Judd (randomly selected from many one file
demos)
----------------------------------------------
polygonamy.unp$8000.prg | 60674| 239|
----------------------------------------------
was freezed with AR then crunched, unpacked/unfreezed file for testing,
$0800-$f4ff, jmp $8000 to start. Was 58blocks.
1st step: exomizer 2.0beta wins
----------------------------------------------
test1.0.exo2.prg | 10205| 41|
test2.0.pu.prg | 12401| 49|
test2.1.pu+delta.prg | 10996| 44|
test3.0.Crushed.prg | 12901| 51|
----------------------------------------------
- Delta packing gives often better results in pucrunch by "flatting"
sequences of progressive values like in sinustables, samples etc
- Crush is run every time with all 6 speeds and only the smallest file
is
kept. Speed 6 is NOT always the best :)
- Exomizer 2.0 gives better results than v1.15 thanks to a new sequence
packing system, can be disabled with option -c and gives more or less
the same sizes of exo v1
Now, let's try to repack them in different combinations.
----------------------------------------------
test1.1.exo2.exo2.prg | 8066| 32|
test1.2.exo2.pu+delta.prg | 7801| 31|
test1.3.exo2.Crushed.prg | 8274| 33|
----------------------------------------------
test2.2.pu+delta.pu+delta.prg | 7457| 30|
test2.3.pu+delta.exo2.prg | 7299| 29| <<< winning combo
test2.4.pu+delta.Crushed.prg | 7499| 30|
----------------------------------------------
test3.1.Crushed.exo2.prg | 9124| 36|
test3.2.Crushed.pu+d.prg | 8631| 34|
----------------------------------------------
Of course there can be files where pu gives better results than exo,
always depends on what is being compressed.
http://iancoog.altervista.org/hid/polygonamy_xcrunch_tests.rar
--
-=[]=---iAN CooG/HokutoForce+TWT---=[]=-
Aaron Bilger
wrote:
>I've made a little test to show some numbers, sizes are in bytes and
>actual blocks occupation.
>
>----------------------------------------------
>test1.0.exo2.prg | 10205| 41|
>test2.0.pu.prg | 12401| 49|
>test2.1.pu+delta.prg | 10996| 44|
>test3.0.Crushed.prg | 12901| 51|
Thanks for the info and the tests. You are the developer of Exomizer?
>Now, let's try to repack them in different combinations.
>----------------------------------------------
>test1.1.exo2.exo2.prg | 8066| 32|
>test1.2.exo2.pu+delta.prg | 7801| 31|
>test1.3.exo2.Crushed.prg | 8274| 33|
>----------------------------------------------
>test2.2.pu+delta.pu+delta.prg | 7457| 30|
>test2.3.pu+delta.exo2.prg | 7299| 29| <<< winning combo
>test2.4.pu+delta.Crushed.prg | 7499| 30|
>----------------------------------------------
>test3.1.Crushed.exo2.prg | 9124| 36|
>test3.2.Crushed.pu+d.prg | 8631| 34|
>----------------------------------------------
Let me see if I am understanding this correctly:
You compressed using each compressor and got to 10-12K.
Taking any of these compressed files, you were able to realize about
another 20-25% compression by using any of the compressors on them
again - including reapplying the same compressor originally used.
Is this correct?
Sorry if the question seems daft, but if that is what you are saying,
I am astounded that any of these leave that much redundancy on the
table to begin with. I am not used to seeing losslessly compressed
output be significantly further compressible except in specialized
cases (eg. RLE then LZ77), and essentially never using the exact same
compressor.
Aaron Bilger
(Ojala Pasi 'Albert') wrote:
>In article <9c09d1ltq5cvpsc79...@4ax.com>,
>As already indicated, Exomizer has beaten pucrunch on compression ratio
>already a couple of years ago.
>
>The difference in compression ratio is not dramatical, and because
>pucrunch can be used for quite many systems, it is still useful.
Thanks. Indeed, though this question is mainly nostalgic curiosity,
the cross-platform C ability of PuCrunch makes me wonder if it could
still be useful in larger systems today. eg. without a 64K
restriction perhaps it could still be useful as a small fast
decompressor on 16/32 bit embedded systems, competitive with the (less
freely usable) LZO.
Aaron
Ojala Pasi 'Albert'
Aaron Bilger <syse...@comcast.net> wrote:
>the cross-platform C ability of PuCrunch makes me wonder if it could
>still be useful in larger systems today. eg. without a 64K
>restriction perhaps it could still be useful as a small fast
>decompressor on 16/32 bit embedded systems, competitive with the (less
>freely usable) LZO.
In fact I'm using a reduced version of pucrunch (RLE is removed) at work
to (de)compress VSDSP code/data. We have an SPI-EEPROM bootloader,
which supports compressed records. Compression is still byte-wise although
the instruction word is 32 bits and data words are 16 bits.
Code generally compresses to 50%, mainly because of unused parallelism
in the 32-bit instruction word, and the C-compiler way of generating
similar code for function prologs and epilogs.
Btw, you can compile pucrunch with -DBIG to go upto 800000 bytes of
output, and even larger by changing the OUT_SIZE define in the source.
-Pasi
--
"Free men can have a need to follow someone, too. Most men want to
belive in something larger than themselves, something wider than
their own fields."
-- Faile in The Wheel of Time:"The Shadow Rising"
Ojala Pasi 'Albert'
Aaron Bilger <syse...@comcast.net> wrote:
>On Wed, 13 Jul 2005 20:34:33 GMT, "iAN CooG" <ian...@despammed.com>
>wrote:
>>test2.2.pu+delta.pu+delta.prg | 7457| 30|
>>test2.3.pu+delta.exo2.prg | 7299| 29| <<< winning combo
>>test2.4.pu+delta.Crushed.prg | 7499| 30|
In this recompression case exomizer has a slight edge because of the
shorter decompression code. You can get some bytes off by -fshort -m6
pu+delta.pu+short-m6.prg 7431 30
For comparison:
-rw-rw-r-- 1 albert staff 11996 Jul 14 11:45 gzip-9.gz
-rw-rw-r-- 1 albert staff 11953 Jul 14 11:45 gzip-9.gzip-9.gz
-rw-rw-r-- 1 albert staff 7150 Jul 14 11:47 pu+delta.gzip-9.prg
The delta compression seems to help a lot..
Note: the gzip versions lack the decompression code.
>another 20-25% compression by using any of the compressors on them
>again - including reapplying the same compressor originally used.
>
>Is this correct?
>
>Sorry if the question seems daft, but if that is what you are saying,
>I am astounded that any of these leave that much redundancy on the
>table to begin with.
The test file is abundantly redundant. I don't know if it was selected
because of that, but the main goal in pucrunch was to target the only
slightly redundant files instead :-)
>I am not used to seeing losslessly compressed
>output be significantly further compressible except in specialized
>cases (eg. RLE then LZ77), and essentially never using the exact same
>compressor.
> head -c 1000000 /dev/zero >file
> gzip -c <file >file.gz
> gzip -c <file.gz >file.gz.gz
> ls -l file*
-rw-rw-r-- 1 albert staff 1000000 Jul 14 11:43 file
-rw-rw-r-- 1 albert staff 1003 Jul 14 11:43 file.gz
-rw-rw-r-- 1 albert staff 64 Jul 14 11:43 file.gz.gz
-Pasi
--
/Nevertheless, it had been a paradise here, in those warm summer days and
nights, when she had sat by the hour talking to Maharet, when she had danced
with Mael and Maharet by the light of the moon. [..] It was to know again
the joy that had gone before./ -- Jesse in "The Queen of the Damned"
Ojala Pasi 'Albert'
iAN CooG <ian...@despammed.com> wrote:
>Pucrunch is quite immediate, Exomizer takes more time to pack, but gives
>satifactions :).
I was developing on a 25MHz 68030 at the time, so I had to take
(and develop) some algorithmic tricks to make pucrunch run reasonably
fast, like 24 seconds instead of the original 28 minutes..
>Of course there can be files where pu gives better results than exo,
>always depends on what is being compressed.
Has anyone tested exomizer on Calgary and Canterbury Corpus test suites?
-Pasi
--
"Science is not about building a body of 'known' facts. It is a method
of asking awkward questions and subjecting them to a reality-check, thus
avoiding the human tendency to believe whatever makes us feel good."
-- "The Science of Discworld"
Aaron Bilger
(Ojala Pasi 'Albert') wrote:
>In article <guebd11kle1tv6ftb...@4ax.com>,
>
>> head -c 1000000 /dev/zero >file
>> gzip -c <file >file.gz
>> gzip -c <file.gz >file.gz.gz
>> ls -l file*
>
>-rw-rw-r-- 1 albert staff 1000000 Jul 14 11:43 file
>-rw-rw-r-- 1 albert staff 1003 Jul 14 11:43 file.gz
>-rw-rw-r-- 1 albert staff 64 Jul 14 11:43 file.gz.gz
*laugh* Yes, I suppose files so redundant they surpass the max run or
sequence length could be further compressed. I didn't realize his
example was highly (though not as much as /dev/zero) redundant.
Aaron
Cameron Kaiser
>-rw-rw-r-- 1 albert staff 1000000 Jul 14 11:43 file
>-rw-rw-r-- 1 albert staff 1003 Jul 14 11:43 file.gz
>-rw-rw-r-- 1 albert staff 64 Jul 14 11:43 file.gz.gz
Um, boggle? You'd think there would be code in there to take advantage of
double compression (or compute when it matters).
Myself, I'm pretty pleased with pucrunch's performance. I wrote a small
assembly tag loader to allow pucrunch to compress and RUN regular BASIC
programs also (along with some Makefile magic to compute lengths and so on).
--
Cameron Kaiser * cka...@floodgap.com * posting with a Commodore 128
personal page: http://www.armory.com/%7Espectre/
** Computer Workshops: games, productivity software and more for C64/128! **
** http://www.armory.com/%7Espectre/cwi/ **
iAN CooG
> Has anyone tested exomizer on Calgary and Canterbury Corpus test
> suites?
No, and i'm not interested on even trying :)
Exomizer (like pu) for me is a C64 data/prg cruncher and not a general
purpose archiver, even if the author has included an ExoRaw.exe on the
latest 2.0 beta to compress pc data>64k.
For that purposes I use Rar/7zip.
--
-=[]=---iAN CooG/HokutoForce+TWT---=[]=-
Line noise? What fh=.hElL is.LinS nfise?
iAN CooG
> The test file is abundantly redundant. I don't know if it was selected
> because of that, but the main goal in pucrunch was to target the only
> slightly redundant files instead :-)
This happened sometimes to me, so I always try double crunching with
both. This was one of the lucky times :)
On C64 we are used to charpack + crunch the programs because native
crunchers works better this way.
Cross crunchers like pu and exo are more flexible and do all the
necessary analisys/packs by themselves, it seems, and there is no need
to charpack, most of the times.
I have not reported the test I've made using charpackers + the 3
crunchers because the final results on pucrunch and exomizer were always
worse than using the crunchers by themselves.
Charpacker tested were No-stack-using/Active, Idiots FX packer, Sledge
hammer 2.2 modded by Fungus.
None of these improved the crunching.
--
-=[]=---iAN CooG/HokutoForce+TWT---=[]=-
I'm trying to find myself. Anyone seen me lately?
iAN CooG
> Thanks for the info and the tests. You are the developer of Exomizer?
No, author is Magnus Lind (Zagon/ex-Vison/ex-Light)
I'm only a betatester and daily user of it, tried to give some
optimizations in the 1.15 release but Magnus completely rewritten it on
2.0, now is even faster than before, and he fixed also some issues on
some test files I reported failing decompression.
--
-=[]=---iAN CooG/HokutoForce+TWT---=[]=-
00-0000SYSTEM ERROR: press F13 to continue.