Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

I have a small toy program...

31 views
Skip to first unread message

jules Gilbert

unread,
Jan 12, 2012, 4:50:42 PM1/12/12
to repeatable_...@yahoo.com
To those who have a business requirement and can prove their identity
and who promise not to further distribute it, I am happy to forward a
small C program I wrote that builds a RAD buffer and then applies it
to zlib, measuring and reporting the degree of compression, buffer
size, etc... The purpose for which I wrote this small program was to
see what zlib was good at, what it's weaknesses are and more
importantly, what it's strengths are.

For example, I learned that a RAD buffer must contain at least a
57-58% bias in order for it to compress at all.

And I found other non-linearities, these useful, ways that knowing my
goods, I could maximize their compressive effect.

Please include a one sentence statement, stating your promise not to
further distribute my program.

Please include the name of the company that you work for or are
otherwise associated with.

I don't distribute zlib. Why?, because every OS I've used in the past
few years has it built in. Even the Apple I use as a mail appliance.
Under unix-like OS's, you merely need to say "-lz" to have the zlib
library loaded.

zlib is not my program, it's the work of Jean-loup and Adler.

Email me at:

repeatable
underscore
compression
ngista
oohay
tod
moc

And please, put the words TOY PROGRAM in the subject so a small mail
tool I use can send it back.

Earl_Colby_Pottinger

unread,
Jan 12, 2012, 8:21:04 PM1/12/12
to
On Jan 12, 4:50 pm, jules Gilbert <jules.sto...@gmail.com> wrote:
> To those who have a business requirement and can prove their identity
> and who promise not to further distribute it, I am happy to forward a
> small C program I wrote that builds a RAD buffer and then applies it
> to zlib, measuring and reporting the degree of compression, buffer
> size, etc...  The purpose for which I wrote this small program was to
> see what zlib was good at, what it's weaknesses are and more
> importantly, what it's strengths are.

Who cares? We already know how to use zlib. Stop wasting time and
write a de-compressor for your compressor and test it.

Jim Leonard

unread,
Jan 13, 2012, 11:22:00 AM1/13/12
to
On Jan 12, 3:50 pm, jules Gilbert <jules.sto...@gmail.com> wrote:
> For example, I learned that a RAD buffer must contain at least a
> 57-58% bias in order for it to compress at all.

The fact that you are using an LZ77/LZSS compressor as the processor
for your residuals shows that you either are unwilling or unable to
comprehend the advice you have been given over the last 1.5 decades.
It also shows a lack of understanding of what zlib is best suited
for. An order-0 statistical back-end will produce better results
(unless your residuals are full of repeating patterns). Try a huffman
encoder instead.

Better yet, stop dicking around and write a decompressor. I'm amazed
you haven't actually tested your decompression method *even once*.
How can you make claims if you haven't actually tested your process?

Thomas Richter

unread,
Jan 13, 2012, 2:03:47 PM1/13/12
to
On 12.01.2012 22:50, jules Gilbert wrote:
> To those who have a business requirement and can prove their identity
> and who promise not to further distribute it, I am happy to forward a
> small C program I wrote that builds a RAD buffer and then applies it
> to zlib, measuring and reporting the degree of compression, buffer
> size, etc... The purpose for which I wrote this small program was to
> see what zlib was good at, what it's weaknesses are and more
> importantly, what it's strengths are.

Jules, you really don't know when to stop, right? No, anyone sane in his
mind will not do business with you. That includes me. The best I can
offer is a free of charge review of your method. I give you a short
answer upfront "it doesn't work". The longer answer will depend on your
program, and it will include why in specific it doesn't work. Again, the
short answer will be "because it is not reversible". The longer version
*where* it is not reversible will depend on your program.

As Jim already stated, zlib is a pretty bad start for zero-order
statistics. Huffman coding might, depending on your alphabet size.
Arithmetic coding might be even better. I posted an arithmetic coder
here a while ago - and you can get a free implementation if you ask for.

[Hint: This is also why PNG doesn't compress very well. Zlib is good for
text or programs, but not for signals.]

> For example, I learned that a RAD buffer must contain at least a
> 57-58% bias in order for it to compress at all.

RAD is not a scientific term. And if the data has a bias, you use the
letter 'R' in 'RAD' for something I would not use it for.

> And I found other non-linearities, these useful, ways that knowing my
> goods, I could maximize their compressive effect.

Nonlinearity is a scientific term, but it does not have a meaning in
compression. It has a meaning in signal processing. If you consider a
compressor as something that converts an input signal to an output
signal, then I have good reasons to believe that there is no compressor
at all that could be called "linear" in any meaning of the word I know.

> Please include a one sentence statement, stating your promise not to
> further distribute my program.

Would you in return provide a one sentence statement to stay away if I
can demonstrate that the program does not work and that you stop bugging
people here?

jules Gilbert

unread,
Jan 14, 2012, 8:13:17 PM1/14/12
to
I've reD what each of you have said here and see that you've made some
wrong assumptions.

First, about zlib. I chose it for several reasons (over other
compressor toolsets.) For one thing, it's very fast!, and my
compressor is fast too, which make's zlib a very good choice. In fact
I've tried several compressors and, because of it's speed, me thinks
zlib is the best choice. Remember my systems are repeatable, it's how
fast you can make a file small that counts.

Second, I doubt if anyone posting her has seen the source code to my
toy program. If you had you would realize why your remarks simply
don't make sense. I said it was a TOY, it's not intended to function
as a production compressor and decompressor; In fact it's not even
intended for experimental use. In fact before I added the copyrights
it was only about 700 bytes. Everything. Seven hundred bytes. (I've
sent out several copies and the accompanying note is larger.)

When I wrote it I was considering which method I wanted to use for a
certain application. After coding it and learning what I needed, I
looked at it and decided to distribute it. Bur I meant what I said
and thus it's very doubtful that any of you guys posting here will
ever see my goods. And I accept that you will always be who you are
now, certainly I can't change you.

This isn't fun or humorous but I have to admit that Richter saying
first that a program he's never seen doesn't exist, then saying it
doesn't work, and then offering to review it, well, in a very sick way
that was a little bit funny.






Thomas Richter

unread,
Jan 15, 2012, 5:28:41 AM1/15/12
to
On 15.01.2012 02:13, jules Gilbert wrote:

> I've reD what each of you have said here and see that you've made some
> wrong assumptions.
>
> First, about zlib. I chose it for several reasons (over other
> compressor toolsets.) For one thing, it's very fast!, and my
> compressor is fast too, which make's zlib a very good choice.

No, it makes a bad choice. Compression performance is, first of all, not
measured in milliseconds, but in compression ratio. Second, it is the
wrong tool for the job, and third there are much better tools that are
faster than ZLib and do the right thing. I already pointed you at the
MQCoder.

> In fact
> I've tried several compressors and, because of it's speed, me thinks
> zlib is the best choice. Remember my systems are repeatable, it's how
> fast you can make a file small that counts.

"This program compresses very bad - but look how fast it is!" I then
suggest a "cp" for optimal performance.(-;

No, really. You need to learn why things compress, and that a model has
to fit to its input. ZLib is not the right program if all you know about
your input is that it is "biased". ZLib is a text/program compressor
that looks for repeating strings in its input, something you do not have
then.

> Second, I doubt if anyone posting her has seen the source code to my
> toy program.

Which does not make any of my arguments invalid. That the image of an
injective function must necessarily be larger than its source holds
regardless of the program.

> If you had you would realize why your remarks simply
> don't make sense. I said it was a TOY, it's not intended to function
> as a production compressor and decompressor; In fact it's not even
> intended for experimental use. In fact before I added the copyrights
> it was only about 700 bytes. Everything. Seven hundred bytes. (I've
> sent out several copies and the accompanying note is larger.)

Look, if you had the guts to finally learn a little bit about what
compression is, and how it works, you would learn immediately that your
claims make neither any sense, but in a much more general way you imagine.

> When I wrote it I was considering which method I wanted to use for a
> certain application. After coding it and learning what I needed, I
> looked at it and decided to distribute it. Bur I meant what I said
> and thus it's very doubtful that any of you guys posting here will
> ever see my goods. And I accept that you will always be who you are
> now, certainly I can't change you.

If that is your opinion, and you are entitled to do so, why don't you
then simply stop posting your nonsense here? You know what to expect,
you resist to learn, and as long as you don't show any sign of
understanding the fundamentals, the reactions will be the same. Instead,
you claim the same nonsense you did before, not showing any evidence.
What else do you expect? In specific, why do you still come here?

> This isn't fun or humorous but I have to admit that Richter saying
> first that a program he's never seen doesn't exist, then saying it
> doesn't work, and then offering to review it, well, in a very sick way
> that was a little bit funny.

No, I have said that it doesn't work as a part of a "compresses
everything" toolchain, which is something else. And you know what: This
is the fun part about mathematics: You *do* know that certain things do
not work, even without seeing them. I do know that a "compresses
everything" program does not exist, and I also do know that I cannot
divide an angle into three equal parts with ruler and compass alone in a
finite number of steps. And it is for the very same reason that 2+2 is
never, ever five, no matter what you want to make me believe, not even
for large values of two and small values of five. It is a contradiction
in logic. All I need is to apply logic to your arguments, sometimes more
steps, sometimes less steps, but the net result is always: There is an
error in your logic.

If you want to know *where precisely* your logic fails, then I need the
program, but only then. In the same vain, I can only say *where* your
angle-division algorithm fails when looking at it - but I do know *that*
it fails, which is a different story.

Mathematics is a very fascinating science, isn't it?

Sebastian

unread,
Jan 15, 2012, 6:32:06 AM1/15/12
to
On 15 Jan., 02:13, jules Gilbert wrote:
> [...] I doubt if anyone posting her has seen the source code to my
> toy program.  If you had you would realize why your remarks simply
> don't make sense. [...]

I doubt if you have read and comprehended data compression text
books. If you had you would realize why your posts simply don't make
sense.

> [...] This isn't fun or humorous but I have to admit that Richter saying
> first that a program he's never seen doesn't exist, then saying it
> doesn't work, and then offering to review it, well, in a very sick way
> that was a little bit funny. [...]

I don't see anything funny about that. I actually find it a little
sad that during the last 15 years you apparently failed to acquire
even a basic understanding of data compression principles. It does
not matter what your programs actually do if the goal you are trying
to reach (or believe to have reached) doesn't make sense.

An analogy:

- "Hey, I have a program with which I can prove that
there is another natural number between 2 and 3."
- "No, you don't! But if you send it to me I'll probably
be able to point out the flaws in the program/proof."
- "How can you say that? You havn't even seen the program!"

The only difference is that when you claim something, it's mostly
really vague and open to interpretation. Possible explanations for
this are (a) weak math skills and/or (b) you're deliberately unclear
to avoid being disproved. And none of these explanations make you
appear like a person who has anything interesting to say about data
compression.


Cheers!
SG

glen herrmannsfeldt

unread,
Jan 15, 2012, 6:53:08 AM1/15/12
to
Thomas Richter <th...@math.tu-berlin.de> wrote:

(snip)
> No, it makes a bad choice. Compression performance is, first of all, not
> measured in milliseconds, but in compression ratio.

That is usual, but maybe not always the best choice.

I can imagine cases where either fast encoding or fast decoding, but
not necessarily both, would be important enough that optimal compression
might be secondary.

As one example, many more backup tapes are written than ever read, and
so fast encoding is fairly important, fast decoding less important.

For document distribution, or for compressed audio or video, one assumes
that it will be encoded once and decoded many times. It may or may not
be necessary to compress in real time, but it is necessary to decompress
in real time.

-- glen

Thomas Richter

unread,
Jan 15, 2012, 7:24:53 AM1/15/12
to
Thanks, I agree, Glen. I just wanted to point out that there are *fast*
compressor schemes available that would work better in such cases than
the zlib, the QCoder in all its variants being one example, and
depending on the alphabet size, Huffman coding being another.

Thanks,
Thomas

glen herrmannsfeldt

unread,
Jan 15, 2012, 8:48:52 AM1/15/12
to
Thomas Richter <th...@math.tu-berlin.de> wrote:

(snip, I wrote)

>> As one example, many more backup tapes are written than ever read, and
>> so fast encoding is fairly important, fast decoding less important.

>> For document distribution, or for compressed audio or video, one assumes
>> that it will be encoded once and decoded many times. It may or may not
>> be necessary to compress in real time, but it is necessary to decompress
>> in real time.

> Thanks, I agree, Glen. I just wanted to point out that there are *fast*
> compressor schemes available that would work better in such cases than
> the zlib, the QCoder in all its variants being one example, and
> depending on the alphabet size, Huffman coding being another.

Yes, I probably shouldn't have added it to the other thread.

I have thought about it various times when not near a computer.
(The first while writing backup tapes, wondering if it
could be faster.)

-- glen

jules Gilbert

unread,
Jan 15, 2012, 1:30:37 PM1/15/12
to
To Jim Leonard: actually Jim, I think being able to reconstruct the
client buffer by exercising a PRNG tops your suggestions. Now my
problem is finding ways to sell the program which keep it out of your
hands (among others,) for years and years. That's my goal now.

And something else, I labeled this note, "A small toy program." Yet
you guys pounced as if I were a fraud.

jules Gilbert

unread,
Jan 15, 2012, 1:43:26 PM1/15/12
to
On Jan 15, 8:48 am, glen herrmannsfeldt <g...@ugcs.caltech.edu> wrote:
Reading all your remarks. I have one more element to add:

What I do to compress and subsequently decompress is quite different
from the kinds of programs you folks write. My work is based on
models that have almost nothing in common with Shannon or Lempel & Zev
or Huffman. Sure, for packaging, to do a lot of things that one must
do I need tools -- and one of them is zlib, Another is argtable. And
I'm still trying to contact the author of another program I'd like to
make use in my program.

I know you guys don't like me -- but I was only trying to offer a
small toy program that maybe someone in an office who makes million
dollar decisions could say to himself "Hey, I'll send this dude an
email and see what he has." But you guys make such decisions
impossible; I can't even give a toy program away. Okay...

But I will sell what I have. And just as I've said, because the
countries you live in allow you to interfere in commerce without
sanction, I won't be making deals where you live. I can't do anything
about you, but I can make certain you don't benefit from my work.


Thomas Richter

unread,
Jan 15, 2012, 2:44:09 PM1/15/12
to
On 15.01.2012 19:43, jules Gilbert wrote:

> What I do to compress and subsequently decompress is quite different
> from the kinds of programs you folks write. My work is based on
> models that have almost nothing in common with Shannon or Lempel& Zev
> or Huffman. Sure, for packaging, to do a lot of things that one must
> do I need tools -- and one of them is zlib, Another is argtable. And
> I'm still trying to contact the author of another program I'd like to
> make use in my program.

This does not invalidate any argument brought up here. Once again: If it
is injective, it must make some streams longer. Whether that is Huffman,
LZ, arithmetic or anything is completely irrelevant for this argument.
It is a completely technology-agnostic argument that is so unbelievably
simple that it is sheer folly to ignore it.

What *is* relevant is that if you have a preprocessor from which you
believe that it creates some biased output you want to compress, then LZ
is not a good algorithm. Hint: It will *not always* create a biased
output if it is reversible. From that follows that if the output is
always biased, it is not reversible. This is also a similarly trivial
homework.

> I know you guys don't like me -- but I was only trying to offer a
> small toy program that maybe someone in an office who makes million
> dollar decisions could say to himself "Hey, I'll send this dude an
> email and see what he has." But you guys make such decisions
> impossible; I can't even give a toy program away. Okay...

Perfect, this will protect at least some people from your "business".
Will you now finally stay away? Thank you.

> But I will sell what I have.

You told for years, yet you didn't. And anyone reading here would at
least be warned.

> And just as I've said, because the
> countries you live in allow you to interfere in commerce without
> sanction, I won't be making deals where you live.

Even better, one fraudster less. Hint: Yes, public authorities do have,
in all countries I'm aware of, the right to interfere with commerce that
is provably a fraud. And trying to sell something that is provably
non-existing *is* called fraud, in all countries I'm aware of.

> I can't do anything
> about you, but I can make certain you don't benefit from my work.

Certainly not. I want to make honest money.

stan

unread,
Jan 15, 2012, 9:31:14 PM1/15/12
to
Thomas Richter wrote:
> On 15.01.2012 02:13, jules Gilbert wrote:
>> I've reD what each of you have said here and see that you've made some
>> wrong assumptions.

Actually, you might have a point here. People have put in way more
time than you deserve trying to understand you. Note to self: don't
assume, even better don't pay any more attention.

>> First, about zlib. I chose it for several reasons (over other
>> compressor toolsets.) For one thing, it's very fast!, and my
>> compressor is fast too, which make's zlib a very good choice.
>
> No, it makes a bad choice. Compression performance is, first of all, not
> measured in milliseconds, but in compression ratio. Second, it is the
> wrong tool for the job, and third there are much better tools that are
> faster than ZLib and do the right thing. I already pointed you at the
> MQCoder.

I would note for the record it's really hard to make a very bad choice
for an imaginary process. It's like my granddaughter reminded me not
long ago while we were playing; if you change your mind and don't want
tea then it's coffee, magically. If it weren't for this pesky reality
thing...

From a more scientific side, if you can gt your compressor to go as
fast as the speed of light it becomes invisible. Maybe you should slow
down Jules.

<snip>

> "This program compresses very bad - but look how fast it is!" I then
> suggest a "cp" for optimal performance.(-;

Speed test: cp vs rm.

<noble effort elided>

> What else do you expect? In specific, why do you still come here?

Your turn Jules.

Noob

unread,
Jan 16, 2012, 4:29:15 AM1/16/12
to
Jim Leonard wrote:

> I'm amazed you haven't actually tested your decompression method
> *even once*. How can you make claims if you haven't actually tested
> your process?

Lemme take a wild guess here.

http://en.wikipedia.org/wiki/Crank_(person)

jules Gilbert

unread,
Jan 16, 2012, 1:16:45 PM1/16/12
to
I had intended to offer a small (70x byte .C program,) to interested
parties. The program compresses RAD input, and yes it uses zlib as
the compression engine. A program that may not be perfect in one or
two ways but you know, when I see large files crunched down to a few
hundred bytes I don't think it matters much.

You and others who post here are used to thinking about compression as
a single step, because you see compression as something that can occur
once, assuming your using a modern, quality tool. Like zlib.

The difference my methods provide is an escape from this "one-time
only" model. I have a method for representing a random buffer very
succinctly and losslessly recreating it. And I use zlib to store the
"keys" necessary to perform the decompression (the recreation.) How?,
well, I will only say that these guys are right, as their programs are
constructed, it's impossible. Not that this gives them the right to
do as they have done, but I acknowledge that the problem was difficult
and that as most people look at the problem, it is indeed unsolvable
-- because no one, no one, violates natural laws.

My program is real, it doesn't make use of hidden files, anything
related to secret wireless transfers, file renaming, hacking the disk
size reporting, or anything that would permit or facilitate a fraud.
And when I show my program I use a floppy, a regular 3.5" floppy to
transfer the final intermediate file (the compressed image,) from one
machine to another.

Oh, and I get, sometimes, about 20:1 compression. And that's from a
single pass. And worst case, about 15:1. And yes, it's this program
that I repeatably apply compression to.

The bad news is that my program operates on fixed-sized blocks of
data. So, assuming that the final final size is one block plus a
fragment, say BLKSIZE/2, then you get (BLKSIZE/15) + (BLKSIZE/2) as
the average final file size.

And before a bunch of you decided to interfere, I was offering a demo
program that does just what I say: It demonstrates that it's possible
to compress RAD buffers using zlib. It demo's this by doing it. In
seven hundred bytes.

An early version, (I wrote this in a few minutes because I needed to
make a choice and to make that choice I needed to know some numbers,)
was defective because it didn't free two buffers at the proper fix-
point (a technical term relating to entry and exit points of blocks of
code in a routine.) Otherwise no one who has this program has
complained.

Jim Leonard

unread,
Jan 16, 2012, 2:08:55 PM1/16/12
to
On Jan 16, 12:16 pm, jules Gilbert <jules.sto...@gmail.com> wrote:
> I had intended to offer a small (70x byte .C program,) to interested
> parties.  The program compresses RAD input, and yes it uses zlib as
> the compression engine.

We don't want your compression program, or any code from you. What we
want is for you to WRITE and TEST the DEcompression phase of your
method. Until you do this, you are a crank.

> I have a method for representing a random buffer very
> succinctly and losslessly recreating it.

No, you have *claimed* you have a method for recreating it. It is now
obvious you have never actually tested your recreation method, which
makes your claim meaningless.

> And before a bunch of you decided to interfere, I was offering a demo
> program that does just what I say:  It demonstrates that it's possible
> to compress RAD buffers using zlib.  It demo's this by doing it.  In
> seven hundred bytes.

Anyone can compress any data. The real trick is decompressing it.
Come back when you can decompress.

Sebastian

unread,
Jan 16, 2012, 6:34:15 PM1/16/12
to
On 16 Jan., 19:16, jules Gilbert wrote:
> On Jan 16, 4:29 am, Noob wrote:
> > Lemme take a wild guess here.
> > http://en.wikipedia.org/wiki/Crank_(person)

:-)

> [...]
> You and others who post here are used to thinking about compression as
> a single step, because you see compression as something that can occur
> once [...]

Please elaborate.

> The difference my methods provide is an escape from this "one-time
> only" model.  I have a method for representing a random buffer very
> succinctly and losslessly recreating it. [...]

Representing something with fewer bits IS compression -- no matter how
many steps you take to do this. In fact, most of the state-of-the-art
compression algorithms -- especially those for video and audio -- are
very smart concatenations of carefully designed modules that are each
performing a very specific task. So, how is what you do any different?
The "black box compressor interface" is exatly the same: there is some
input and there is some output. And in the case of lossless
compression, there has to be another black box which takes the output
of the previous box and turns it into the original file without any
hidden communication between the black boxes. If you claim that your
method is worth looking at, it has to have such a interface on the
surface (no matter what happens inside). And if there is a
mathematical proof that applies to all such black boxes, it applies to
your method as well since such proof would not exploit any
implementation details. We call them "black" boxes because we cannot
look inside. There is no escape from such a proof unless you change
the rules. And if you change the rules you're basically cheating.

Cheers!
SG

stan

unread,
Jan 17, 2012, 9:07:05 PM1/17/12
to
jules Gilbert wrote:
>
> I had intended to offer a small (70x byte .C program,) to interested
> parties. The program compresses RAD input, and yes it uses zlib as
> the compression engine. A program that may not be perfect in one or
> two ways but you know, when I see large files crunched down to a few
> hundred bytes I don't think it matters much.

To use the term "compression" you must be able to decompress,
otherwise you are simply shrinking data into a small mess. As long as
you never decompress you can easily shrink any file into 1 bit in a
single pass.

> You and others who post here are used to thinking about compression as
> a single step, because you see compression as something that can occur
> once, assuming your using a modern, quality tool. Like zlib.

You have repeatedly refused to make any real effort to understand
any position other than your own. The above statement is a point of
evidence. To be very clear, the number of steps in your process is in
no way relevant to the validity of your claims.

> The difference my methods provide is an escape from this "one-time
> only" model. I have a method for representing a random buffer very
> succinctly and losslessly recreating it.

THIS is your repeated bogus claim, and your idea of preprocessing the
data has been considered and studied thoroughly by some pretty smart
people.

<snip>

> but I acknowledge that the problem was difficult and that as most
> people look at the problem, it is indeed unsolvable -- because no
> one, no one, violates natural laws.

The problem, your claim, is not "difficult" it is one of those natural
laws. The problem seems to be that while nobody violates these laws
they are apparently beyond the grasp of some people. People who fail
to understand frequently find apparent loopholes to get around the
problem only to discover when the rubber hits the road that their
loophole is in fact a pothole.

> My program is real,

Your claim is real, there is no evidence the program is real and you
have yet to actually claim you can reverse the process.

> The bad news is that my program operates on fixed-sized blocks of
> data.

I'm afraid the bad news isn't really that limited.

> And before a bunch of you decided to interfere,

Strange you seem to define refusal as interference. You came and made
the same claims as before and got the same response, to expect
otherwise is at best unwise.

jules Gilbert

unread,
Jan 19, 2012, 8:27:52 PM1/19/12
to
Actually, for maybe a decade I did try, and very earnestly, to make
friends here. And yes, with a few people, maybe five, I did. But I
have gotten attacked by, what?, perhaps as many as 300 people. This
estimate is probably wrong -- I have no idea of a more accurate
estimate. (It's above 200, I'm pretty sure.)

So now we fight on and of course I am determined not to budge -- For
example, I recently responded with a letter to a prospect and I made
certain he understood that he could only use my program in certain
countries.

I do have very different ideas about compression, that's true.

Now to you personally, I have to put some timing support in my
program, maybe you can help me...

I want to time everything, which means four components; My
compression pre-process and decompression post-process, and also the
zlib compression and decompression steps. Showing these times info is
probably a good idea. Now, I've got people who might know a little
bit (I'm not sure,) who have commented here, but what they don't know
is that I've got several types of compressors (three types,) and one
method clearly violate those premises that most people who post here
here hold so dear. Others, those based on the data transfer model,
obey all usual the traffic laws; Unfortunately those programs are my
worst performers.

My point: Pretty much I am content to let these guys get agitated.
It used to bother me, not now. At one time I was hoping to sell my
program for $100M or so and while that could still happen, I've been
in contact with several players at some pretty big companies and they
say that they are turning down deals that once were considered to be
good business opportunities. So, I'm looking to sell into countries
like China and elsewhere.

Oh, more about some needed extensions...

I use argtable, so I will probably plug in a rich set of options.
Important, I want to be able to show BOTH the compression times and
the decompression times. And I will probably soon have to do files as
segments, wherein I process a large block, say 10MB, completely, then
continue for all remaining blocks.

Jim Leonard

unread,
Jan 20, 2012, 10:36:25 AM1/20/12
to
On Jan 19, 7:27 pm, jules Gilbert <jules.sto...@gmail.com> wrote:
> I use argtable, so I will probably plug in a rich set of options.
> Important, I want to be able to show BOTH the compression times and
> the decompression times.  And I will probably soon have to do files as
> segments, wherein I process a large block, say 10MB, completely, then
> continue for all remaining blocks.

Why are you concerned about timing your code when you haven't verified
your decompression process works? Shouldn't you be verifying that
first?

jules Gilbert

unread,
Jan 20, 2012, 10:05:13 PM1/20/12
to
Some basic numbers.

A file below about 550 bytes won't compress. I don't know the actual
numeric lower limit, because a specific number doesn't exist, of
course. The process performance depends on the exact content of
course. But I have files around that size that, when zlib tries, get
larger. But 600 byte files do indeed compress, just not a lot.

A file of 16k will drop by about 7%, per pass.

A file of 50k will drop by about 11.5% or so.

And a file of about 60k will drop by about 12% or so.





Thomas Richter

unread,
Jan 21, 2012, 5:02:17 AM1/21/12
to
On 21.01.2012 04:05, jules Gilbert wrote:
> On Jan 20, 10:36 am, Jim Leonard<mobyga...@gmail.com> wrote:
>> On Jan 19, 7:27 pm, jules Gilbert<jules.sto...@gmail.com> wrote:
>>
>>> I use argtable, so I will probably plug in a rich set of options.
>>> Important, I want to be able to show BOTH the compression times and
>>> the decompression times. And I will probably soon have to do files as
>>> segments, wherein I process a large block, say 10MB, completely, then
>>> continue for all remaining blocks.
>>
>> Why are you concerned about timing your code when you haven't verified
>> your decompression process works? Shouldn't you be verifying that
>> first?
>
> Some basic numbers.

85.6% of all numbers are made up.

> A file below about 550 bytes won't compress. I don't know the actual
> numeric lower limit, because a specific number doesn't exist, of
> course. The process performance depends on the exact content of
> course. But I have files around that size that, when zlib tries, get
> larger. But 600 byte files do indeed compress, just not a lot.
>
> A file of 16k will drop by about 7%, per pass.
>
> A file of 50k will drop by about 11.5% or so.
>
> And a file of about 60k will drop by about 12% or so.

And will it decompress again to its original form? No? Not tested, not
yet implemented after 20 years of work? Oh, too bad.



Sebastian

unread,
Jan 21, 2012, 4:55:49 AM1/21/12
to
On 21 Jan., 04:05, jules Gilbert wrote:
>
> Some basic numbers.
>
> A file below about 550 bytes won't compress.  I don't know the actual
> numeric lower limit, because a specific number doesn't exist, of
> course. The process performance depends on the exact content of
> course.  But I have files around that size that, when zlib tries, get
> larger.  But 600 byte files do indeed compress, just not a lot.
>
> A file of 16k will drop by about 7%, per pass.
>
> A file of 50k will drop by about 11.5% or so.
>
> And a file of about 60k will drop by about 12% or so.

These numbers strongly suggest that you're lying and/or that your
"preprocessing" module maps different inputs to the same output
(possibly due to one or more bugs in your software).

glen herrmannsfeldt

unread,
Jan 21, 2012, 9:34:42 AM1/21/12
to
Jim Leonard <moby...@gmail.com> wrote:

(snip)
> Why are you concerned about timing your code when you haven't verified
> your decompression process works? Shouldn't you be verifying that
> first?

One of the fun things you learn while writing a compressor for an
existing decompressor is the ways things can go wrong.

As noted previously, I am wrote a JBIG2 compressor, using Adobe reader
to test it. Recently, after adding the ability to move the movable
pixels in the context template, I thought it was working.

It turned out to work for about 99 out of 100 pages that I tried it on.
There weren't that many places to go wrong, and it didn't take long to
find, but I can imagine how much harder it is when you are debugging
both the compressor and decompressor at the same time.

It turned out that I got the logic that says pixels outside the actual
image should be zero wrong, such that they would wrap to the next line.
Most pages have white on the left margin, but, it seems, a small number
didn't.

I presume that Adobe has well tested their implementation and that
all bugs are mine, but one can never be sure. There can easily be
corner cases that occur very rarely, and that real data has a
tendency to avoid.

-- glen

Jim Leonard

unread,
Jan 23, 2012, 2:09:29 PM1/23/12
to
On Jan 20, 9:05 pm, jules Gilbert <jules.sto...@gmail.com> wrote:
> But 600 byte files do indeed compress, just not a lot.

They decompress too? You've tested it?

jules Gilbert

unread,
Jan 23, 2012, 10:18:37 PM1/23/12
to
As I recall the routine name is "uncompress" with the same arguments
as the compress function.

pfraser

unread,
Jan 23, 2012, 10:53:33 PM1/23/12
to
jules Gilbert wrote:

> As I recall the routine name is "uncompress" with the same arguments
> as the compress function.

Does it work?
i.e., will it take everything that your compressor generates,
and uncompress it to a file identical to the initial file?

glen herrmannsfeldt

unread,
Jan 24, 2012, 6:33:02 AM1/24/12
to
Especially the very unusual or rare cases? I wrote previously
about a problem in my jbig2 code that showed up in about one
in one hundred tries, using real scanned files.

As compressors get more complicated, there may be many conditions
that show up very rarely. There are systematic ways to track down
such problems, but not as much fun as writing program, and so
it is rare that people do it.

-- glen

Earl_Colby_Pottinger

unread,
Jan 24, 2012, 9:30:57 AM1/24/12
to
You will notice that no matter how you ask or how many times you ask
you will never get him to commit to saying that he has successfully
decompress any data that he had compressed and then do a byte-to-byte
comparison to ensure that it was properly decompressed.

In the past he has refused to have others test his system under
conditions that would prevent a scam (ie. two computer test,
inspecting the computer for hidden storage/communication devices,
booting off clean OS drives ... etc.).

He may have started off as a compression kook, but lately all he seems
to want is money without proof that he has a working system.

PS. I also noticed that the way he talks about Non-Disclosure
contracts implies that he thinks if you sign one that it prevents you
from announcing if his system fails any of the tests you did with it.
Protecting 'theory of operation' or 'program code' with a ND, ok.
Protecting information on the system's failure, no way.

Jim Leonard

unread,
Jan 24, 2012, 10:31:41 AM1/24/12
to
On Jan 23, 9:18 pm, jules Gilbert <jules.sto...@gmail.com> wrote:
> > > But 600 byte files do indeed compress, just not a lot.
>
> > They decompress too?  You've tested it?
>
> As I recall the routine name is "uncompress" with the same arguments
> as the compress function.

You know what I meant. You are dodging the question, which means the
answer is "no", which means you do not have a working system, which
means you cannot make any claims. Either write and test your
uncompressor, or ask for help in understanding how your system is
flawed.

You claim to be intelligent; you also claim to be a decent Christian.
You are not acting like either.

glen herrmannsfeldt

unread,
Jan 24, 2012, 10:35:44 AM1/24/12
to
Earl_Colby_Pottinger <earlcolby...@sympatico.ca> wrote:

(snip)
> PS. I also noticed that the way he talks about Non-Disclosure
> contracts implies that he thinks if you sign one that it prevents you
> from announcing if his system fails any of the tests you did with it.
> Protecting 'theory of operation' or 'program code' with a ND, ok.
> Protecting information on the system's failure, no way.

As far as I know, you can write NDAs that way, and some do.

Then again, IANAL.

-- glen

pfraser

unread,
Jan 24, 2012, 10:55:40 AM1/24/12
to
Earl_Colby_Pottinger wrote:

>
> He may have started off as a compression kook, but lately all he seems
> to want is money without proof that he has a working system.

I think he's lonely, and we're his "friends".

Pete

Earl_Colby_Pottinger

unread,
Jan 24, 2012, 4:00:19 PM1/24/12
to
On Jan 24, 10:35 am, glen herrmannsfeldt <g...@ugcs.caltech.edu>
wrote:

> As far as I know, you can write NDAs that way, and some do.

But after I read such a requirement in a NDA from someone I did not
trust I would never sign it.

Keeping his methods secret - no problem.

Keeping his success/failure secret - no way.

jacko

unread,
Jan 24, 2012, 4:37:55 PM1/24/12
to
Although I hate to agree with Earl the whirl, making an over restrictive NDA is like silly. In fact an NDA serves no purpose for such a revolutionary technology. It's not as though the banks haven't ripped almost everyone off anyhow, and making paper to protect money futures is the wrong way. Just insure against it, in a manor similar to credit default swaps. They may give you long odds on the sucess part, and so get good coverage for almost free.

Cheers Jacko.

jules Gilbert

unread,
Jan 28, 2012, 12:22:05 AM1/28/12
to
On Jan 21, 9:34 am, glen herrmannsfeldt <g...@ugcs.caltech.edu> wrote:
Their is so much mis-information in these posts. I am awe struck at
the wrong-headedness.

First, I would have to trust someone before allowing them to sign an
NDA. And, for various reasons no one who posts questions and
discussions will be asked to sign an NDA. It's not you (well, not for
most of you.) So don't think I'm attacking you or disrespecting you.
Absolutely not. it's just not in my business interests, nothing
personal.

Oh, and another reason, (very practical!,) I don't have an NDA
prepared. Very simply such a document doesn't exist. The people I
deal with just get source code and I would never ask them to sign such
a document, it would just be a terrible insult. No thank you!! (I
only deliver source code, in "another life" -- an expression, I used
to distribute load modules and .EXE's. Now, only C source code.)

Second, every single person who posts here can sleep soundly, as I
will never ask you to sign my NDA. Relax. I have a few friends here
but unless I'm about to get a Fed-Ex letter inviting me to negotiate
or (maybe!,) a buyout letter, I have no expectation that I have to put
such a document together. Again, relax...

Third, about my simple 700 byte program. It used the output of a PRNG
to fill a buffer. The density of the buffer was controlled by a value
from 50 to 99, which designated the bit density ratio. (Oh, that 700
bytes of C source turns into about 1100 bytes with the copyright's and
stuff. And the email accompanying my attached source had a few
notes. Do I count that text as well?)

Fourth, I put this simple program together so that people I'm talking
to will realize that random data CAN BE COMPRESSED. Of course data,
at say, 60% density doesn't compress very well, (with zlib that's less
than one percent, in fact it's nearly zero,) that was another point my
program demonstrated. Except my programs don't deliver 60% data to
the zlib "compress" buffer, the data presented is a minimum of 80%.
(Yes. I can turn client RAD from 50% dense to 80% dense without
storing additional information. Isn't that nice.)

And fifth, I showed my best CLASS2 program to someone today. He had
some criticisms. So here it is... I have at least a couple more
week's of work to do before I can say I'm really done. And I know
this tune, really, I'm still months away from a completely finished
program. But this is the method that has the potential to do
incredible things, the other two classes are just compressors.
CLASS2?, information theory was never so much fun!

jules Gilbert

unread,
Jan 28, 2012, 12:30:47 AM1/28/12
to
No, Sebastian -- I'm not lying. Not one bit!!, these numbers were
written by me several days ago with me looking at one screen and
typing into my Apple, which I use as my internet appliance.

I have a directory of 9,998 gzipped files, named ordinally. The files
range in size from about 1024 bytes to about 256k bytes. And I have a
centralized research and test engine that gets a number from a PRNG
and uses that to "pick" a file to process. (I have no control over
which files get chosen, nor do I know the type of contents, though
most are not things like MPEG or JPEG files. And probably I should
have more bzip based files.)

I was looking at a simple report as I wrote those numbers. reading
off of one screen and writing on my apple. Okay?

Sebastian

unread,
Jan 28, 2012, 9:59:06 AM1/28/12
to
On 28 Jan., 06:30, jules Gilbert wrote:
> On Jan 21, 4:55 am, Sebastian wrote:

> > These numbers strongly suggest that you're lying and/or that your
> > "preprocessing" module maps different inputs to the same output
> > (possibly due to one or more bugs in your software).
>
> No, Sebastian -- I'm not lying.  Not one bit!!

I didn't say that you are lying. I said that lying is one possible
explanation for the numbers.

pfraser

unread,
Jan 28, 2012, 10:00:16 AM1/28/12
to
jules Gilbert wrote:

> I have at least a couple more
> week's of work to do before I can say I'm really done.

You've been saying that for over fifteen years now.

Pete

jules Gilbert

unread,
Jan 28, 2012, 2:06:27 PM1/28/12
to
Well Pete, get comfortable!, because I'm doing all I can. I wrote you
another note. Please read it.

jules Gilbert

unread,
Jan 28, 2012, 2:03:50 PM1/28/12
to
No Pete, you've got it wrong. When I started I didn't know as much as
I wanted, and several of you knew quite a bit more than me about this
thing we call "compression" (really, it's about the transfer of
information, and more importantly, the non-transfer.)

But now I have three different classes of systems; One of which has
implications beyond compression.

And Peter, as I recall, several years ago I corresponded with you,
attempting to befriend you, but you declined. Okay.

I don't think your interest in my social life is healthy, I'll tell
you this much: I'm older than most of you and have a stable set of
friends.

What I lack is a plan for going forward. I have investors who want
cash out. So do I. But none of us are willing to take a loss on the
years or the associated costs. And above all we don't want to see the
programs (or underlying methods,) wind up being used without
appropriate compensation.

As you might be able to infer from my posts here, I am double minded
on this matter. If I never cash out I will still sleep at night. I
did the work I promised, and my investors recognized the all too real
peril of technology investments. They read the articles about Bill
Gates' memory loss and began to realize that what I had been telling
them was true. That the large corporate players were/are exceedingly
corrupt. I explained that a US hitech firm would ask to sign our NDA,
promise us everything, ignore their promises, and then, probably with
great gusto, say "So sue us!" when we complained.

So we're looking further afield. To opportunities where we're not
expected to trust.

And Peter, as for them trusting us (or me!) It's very simple, we have
an application note. Either we meet those spec's or I've committed
fraud.

My responsibility is to show that my programs are in compliance with
the specifications of my applications note. If I can't do that, and
do it routinely and repeatably, it's fraud.

(Oh, and I charge for demo's. I have to pay for policemen. It's not
free for me to do a demo and my health isn't great, a one day demo
means I won't be doing anything else that week.)

I show my process working with any client supplied content. (How can
I promise "any?" Because the first thing my programs do is to "jazz"
the content, we apply a stage of randomization, so I really don't care
too much. Yes, the file statistics do change somewhat. If someone
were to give me a file of 1's, then I'm compressing the output of my
very own PRNG. So far that's not happened. That wouldn't be fair (to
us,) and I probably wouldn't do business with someone who did that.)

So, what does my app note say?, it's lengthy but one value statement
is this: That we can, without storing any additional data or keys or
results of any kind, convert 50:50 data to 80:20 data and
subsequently, losslessly, reverse the process.

And I assert, quite truthfully, that the process is C based, that the
program is fast and uses almost no memory overhead.

And as everyone who comes here knows, mostly because of me, data that
is less than about 60% won't compress, but 80% data will. Yes, if you
want 2:1 using zlib you need 95% data. But 80% bit density works for
me.

So, if someone is willing to play by my rules I expect we can do
business.

Today I have two methods that I like.

Jim Leonard

unread,
Jan 30, 2012, 1:01:38 PM1/30/12
to
On Jan 28, 1:03 pm, jules Gilbert <jules.sto...@gmail.com> wrote:
> So, what does my app note say?, it's lengthy but one value statement
> is this:  That we can, without storing any additional data or keys or
> results of any kind, convert 50:50 data to 80:20 data and
> subsequently, losslessly, reverse the process.
>
> And as everyone who comes here knows, mostly because of me, data that
> is less than about 60% won't compress, but 80% data will.  Yes, if you
> want 2:1 using zlib you need 95% data.  But 80% bit density works for
> me.

Protip: It would make you seem less like the crank you are if you used
common terminology. "80:20 data" and "80% data" are undefined,
meaningless terms. (No, I don't want you to define them.)

> So, if someone is willing to play by my rules I expect we can do
> business.

I hope you never actually get this far because it will destroy you
financially. When you cannot deliver a system that does what you
claim, you will be sued for fraud, and one of the last things you want
in your golden years is a lawsuit.

stan

unread,
Jan 30, 2012, 6:59:15 PM1/30/12
to
Jim Leonard wrote:
> On Jan 28, 1:03 pm, jules Gilbert <jules.sto...@gmail.com> wrote:
>> So, what does my app note say?

By your own admission, we will never know.

>> And as everyone who comes here knows, mostly because of me, data that
>> is less than about 60% won't compress, but 80% data will.

No one here knows anything because of you. You have refused to discuss
your method for reasons of your own and you likewise have never
demonstrated anything. You unbelievable claims haven't educated anyone.

> Protip: It would make you seem less like the crank you are if you used
> common terminology. "80:20 data" and "80% data" are undefined,
> meaningless terms. (No, I don't want you to define them.)

80% imaginary method will reduce a file to ....
20% chance of ever getting your data back...
0 new messages