Compression update: I failed! :(

Harry Potter

unread,

Apr 28, 2012, 10:42:44 AM4/28/12

to

As some of you migt already know, I've been working on file compression or some time now. I' been working hard and tweaking it every way possible to no avail. My first goal was to produce the best general-purpose lossless technique available: first better than PKZip' deflate using teak, then better than RAR using arithmetic coding then, finally better than 7-Zip using everything I can grab. I planned to compress _everything_, including FAT32 hard drives on Win98--even if the compressed drive can only be FAT16--, my own version of WinMount with write access and a portable driver, executable compression, emulation, archives and back-ups and eventually multimedia. I even tried several signature techniques. At first they seemed good, but when I debugged them, they back-fired. I will not ask for help; I believe that is my job. I am just confessing it here.

Ernst

unread,

Apr 28, 2012, 5:19:08 PM4/28/12

to

On Saturday, April 28, 2012 7:42:44 AM UTC-7, Harry Potter wrote:
I am just confessing it here.

Ah, yeah...

We all have been there in some form.

So, get back up, dust yourself off and get back on the horse.

I'm at some coding today myself and I do not know it it will be a success or a failure once this project is able to run. That's the risks and the reward is a learn new things.

Good luck with your efforts and feel free to chat if you need it.
I spend so much time working on these projects that it helps to chat with someone about things even is an abstract and disjointed way.

Harry Potter

unread,

Apr 30, 2012, 9:01:56 AM4/30/12

to

Thank you, you've been very supportive. I have an idea to do better, but it'svery complex. Maybe I'll get back to it later....

Ernst

unread,

May 20, 2012, 8:54:27 PM5/20/12

to

Well, it takes time but if this offers you an interest in life then it isn't a bad thing.

Sometimes I find I step right up to a big problem and face it down. The worst is always a "draw"; where there are no winners or losers.
If we learn something then there is no loss.

I will say that I find that I am building on past experiences more and more each time out!
For example I am now indexing the Million Digit file in a matrix generated out of a 10kbyte program.
The idea is to now hone my efforts of Hashing.

If there is to be any solution then a 1677721.6 fold pay out for the "dictionary" is a good start. I mean that for 10k I get 1677721.6 times the size in utility. In itself that is compression of a large order.

So you may find that you will revisit many truths over and over when looking at a problem from different points of view! I do. Yet it is important to review even if it seems a waste of time! You never know when a little time gone by will open your eyes wider!

Well I will check back from time to time.. As always.

Good luck on your projects!

Harry Potter

unread,

May 21, 2012, 10:04:06 AM5/21/12

to

Ernst, I thank you for your support. :) Maybe I'll revisit the problem later.

Ernst

unread,

May 29, 2012, 11:44:23 PM5/29/12

to

Never give up.. Never surrender!

Every step I have taken, Every line of code I have written, has been the foundation of today's success.

Someday you will "try again" unless you have another dream or child support.

Be Well..

Harry Potter

unread,

Jun 7, 2012, 10:46:06 AM6/7/12

to

I picked up on file compression again. Debugging is attrocious! :/ But I have ideas...again!

Ernst

unread,

Jun 16, 2012, 12:56:49 AM6/16/12

to

Sounds good..

What Advice I can give is follow your life-interests.. Win of lose we all die so do what feels right.

I'm still looking for a way to encode "random data."

Generating a 16GB data set within a 10k program hasn't given me an advantage.

I will now revisit the original effort which guided me in this direction with these experiences working the dataset.

What seems to be true is we evolve; not as a species but as a person. We have brains that develop into mature and we then taper off as we age.. It's a bell shaped curve..

Enjoy the impus of being it's all we have.

So what language? If I am ask.

Harry Potter

unread,

Jun 17, 2012, 6:58:03 AM6/17/12

to

On Saturday, June 16, 2012 12:56:49 AM UTC-4, Ernst wrote:
> I'm still looking for a way to encode "random data."
>

Didn't find it. I'm sorry for making that claim. I was cocky! :(

> What seems to be true is we evolve; not as a species but as a person. We have brains that develop into mature and we then taper off as we age.. It's a bell shaped curve..
>

That's certainly true.

> Enjoy the impus of being it's all we have.
>
> So what language? If I am ask.

Right now, mostly C. When I'm finished, I will convert most of the code to assembler and probably use Open Watcom to create the Win32 versions.

Ernst

unread,

Jun 17, 2012, 4:42:53 PM6/17/12

to

C here myself..

C is all I have needed.

I am now merging two branches of one project. Sometimes I need to go sideways before I can go forward.
Then it's all about indexing and hashing. Still unknown if it will work.

Hey it's been 100+ 5 days in a row.. Freaky June here.

Ernst

unread,

Jul 16, 2012, 6:39:00 PM7/16/12

to

Hey .. How is it going?

Things are on the positive side here for a change. I have a potential solution to the Million Digit Challenge. It's going to take a while to do the "brute force" searching but it looks to be a solution.
It's not a magic function so there are real boundaries like any other maths.

I wrote a quick decoder to test with and that went well so today is about writing a complete decoder. One that is very small is size for sure.

So hey, I hope you are still around. Drop a reply if you see this.

Ernst

Harry Potter

unread,

Jul 17, 2012, 8:44:26 AM7/17/12

to

I'm still working on file compression. It is slow work, :( but I'm getting there. I'm working on the first step of compression. Right now, I'm using a signature technique, and, along with Huffman and LZW (Did I get it right: to shorten a repeated lock to a reference to the last occurence?), it seems to do slightly better than deflate. I wasn't going to reveal anything until this technique is debugged. And, even when I do debug it, I'll need to keep improving it, i.e. make Huffman adaptive, switch to arithmetic coding, try another signature technique, do more tweaks (they help a couple hundred bytes here and there...), etc. Thank you for showing interest.

On Monday, July 16, 2012 6:39:00 PM UTC-4, Ernst wrote:
> Hey .. How is it going?
>

> Things are on the positive side here for a change. I have a potential solution to the Million Digit Challenge. It's going to take a while to do the "brute force" searching but it looks to be a solution.
> It's not a magic function so there are real boundaries like any other maths.

Ernst

unread,

Jul 17, 2012, 4:26:47 PM7/17/12

to

On Tuesday, July 17, 2012 5:44:26 AM UTC-7, Harry Potter wrote:

How strange, There are text editor codes showing up in the quotes.. Anyway

Wow your work is technical! Signature method? I'll look that up.

Approaching the first 1000 "encodings" here. was at 659 when I logged in. Also there are three kittens here too so it's a rather pleasant time for us all.

What do you think of this method? http://en.wikipedia.org/wiki/Context-adaptive_binary_arithmetic_coding#Algorithm

I've been looking into context schemes.

Just looked up that signature technique and that is a great pdf yeah.. I'll read that!

Harry Potter

unread,

Jul 18, 2012, 8:57:03 AM7/18/12

to

On Tuesday, July 17, 2012 4:26:47 PM UTC-4, Ernst wrote:
> Wow your work is technical! Signature method? I'll look that up.
>

By signature technique I mean my own technique, one that I created myself. Unfortunately, it only gave me a couple % extra compression ratio. :(

> What do you think of this method? http://en.wikipedia.org/wiki/Context-adaptive_binary_arithmetic_coding#Algorithm
>

I'll look at that later, when I'm at that point.

Ernst

unread,

Aug 26, 2012, 6:51:21 PM8/26/12

to

On Saturday, April 28, 2012 7:42:44 AM UTC-7, Harry Potter wrote:
> As some of you migt already know, I've been working on file compression or some time now. I' been working hard and tweaking it every way possible to no avail. My first goal was to produce the best general-purpose lossless technique available: first better than PKZip' deflate using teak, then better than RAR using arithmetic coding then, finally better than 7-Zip using everything I can grab. I planned to compress _everything_, including FAT32 hard drives on Win98--even if the compressed drive can only be FAT16--, my own version of WinMount with write access and a portable driver, executable compression, emulation, archives and back-ups and eventually multimedia. I even tried several signature techniques. At first they seemed good, but when I debugged them, they back-fired. I will not ask for help; I believe that is my job. I am just confessing it here.

The nature of information; and indeed that which all of us and all thing are is the challenge.

We can be no more that what makes us possible.

The least is we understand what is "real" to us.

Good Luck Man!