Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Random data compression and PI

169 views
Skip to first unread message

Christopher Smith

unread,
Feb 3, 2012, 5:30:39 AM2/3/12
to
Hello all,

I promised myself I wouldn't write in here unless I absolutely thought I had a decent idea, and I now find myself needing some help. Theres alot here, so please follow along carefully, and if there is a better way I can explain something I will edit the post.

(All computational time aside)

Take 10 characters, or 80 bits.
Each character(or 8 bits) has an identity, 000/255 to 255/255.
Group(not add) these identities to form a 30 digit number.
Split this number into 3, 10 digit sections. Add leading zeros if necessary.
The highest number possible in each section would be 2552552552

Example:
Section-1:0256254824
Section-2:1100225544
Section-3:2552552552

Now expand PI to 2,552,552,557 decimal places (2552552552+5, will explain 5)
This can be done rather quickly using an efficient algorithm. I will use y-Cruncher to build a one time file soon. It does not need to be built every time we run this, only when the data does not exist.

The values I will use below are EXAMPLES ONLY, the true values are in PI!

Section-1(or 2 or 3) becomes Index-1. Index-1 represents a position in the digits of Pi.

At Index-1, the 4 digits in Pi could be 9503, lets call this the Lead-Number.

Using the Lead-Number, build 4 more digits by grabbing a number at each numerical position in Index-1 (the first digit is position Zero). Lets call this the Tail-Number.

(0256254824:9503)

9 is 4
5 is 5
0 is 0
3 is 6

Combine to build a SemiFinal-Number: 95034506

Search Pi for the Lead-Number.
Each numerical position that the Lead-Number is found at is called Temp-Index.
Add leading zeros to Temp-Index if its length is less than 10 digits.
Check positions 9,5,0, and 3 inside the Temp-Index, and call this value Temp-Tail. Combine Lead-Number and Temp-Tail to create Temp-Number

Each time Lead-Number is found and Temp-Number matches the SemiFinal-Number, add 1 to a variable called Iterations, starting with a value of Zero.

Each time Lead-Number is found, Temp-Number matches the SemiFinal-Number, And Index-1 matches the Temp-Number, Halt.

Take the 8 numbers and convert them to 4 bit binary, then combine to create a 4 Character long SemiFinal-Key: � E

Make Iterations a 3 digit number by adding leading zeros if needed.
Convert this 3 digit number into a character (000/255 to 255/255, IE if iterations are 104, then the character would be h).
Combine the SemiFinal-Key and Iterations to form the Final-Key: � E h

If Iterations ever reach > 256 for all indexes 0-2552552552, we will require a few Iteration multipliers, we can use 2 multipliers to get a value above the maximum digits of Pi, and still hold compression.

This could be incredible, as it will say that data of 80 bits of data can be extracted from Pi using 40 bits of information with no multiplier, 48 bits with 1 multiplier, and 56 bits with 2 multipliers.

I need help programming this. I will start working on it today by building the digits of Pi with y-Crunch.

Christopher Smith

unread,
Feb 3, 2012, 6:05:49 AM2/3/12
to
Edit 1: "Take the 8 numbers" refers to the SemiFinal-Number

Edit 2:
"If Iterations ever reach > 256 for all indexes 0-2552552552"
should read
"If Iterations ever reach > 256 for any indexes 0-2552552552"

Edit 2: Let me explain the "80 bits of data" I speak of. The information extracted from Pi would be a 10 digit position, and thus only 10/30 original numbers that we computed in the beginning. We would ultimately have to extract 3, 10 digit positions to recreate 30/30 numbers, our initial 10 characters.

Christopher Smith

unread,
Feb 3, 2012, 6:25:55 AM2/3/12
to
Err: I thought I was on to something until I read the last portion to myself again.

It would be 10 digits per 40 bits, however we need 30 digits. That means 120 bits

*smack in head*

Lemme figure a few things out Ill be back..

Sebastian

unread,
Feb 3, 2012, 7:50:29 AM2/3/12
to
On 3 Feb., 11:30, Christopher Smith wrote:
> [...]

I just skimmed it, but it looks like the "magic function theory"
approach where instead of a pseudo random number generator you intent
to use digits of PI.

http://www.dogma.net/markn/FAQ.html#Q19

Anyhow... I did not find any description of what this method is
supposed to be good for. You should lead with that. Chances are that
you're trying to do something that is known to be impossible or at
least so hard that it is practically impossible. In that case you
could save yourself a lot of hassle by directing your attention to
more "interesting" problems.

Cheers!
SG

Ernst

unread,
Feb 3, 2012, 10:15:23 PM2/3/12
to
If you don't mind what do you think of Pi or irrational numbers in
general?

I find bit patterns are interesting.

Forget about the goof. I am interested in your take on Pi. It's
what? http://en.wikipedia.org/wiki/Pi

Ernst

Christopher Smith

unread,
Feb 4, 2012, 1:13:35 AM2/4/12
to
Doing the impossible is interesting, which is why I try :) Its been proven impossible only as of now. If it were to be done, humans would be on another conquest to understand more about why rules can be broken. I believe all information exists in Pi, or any transcendental number, and I am looking for a better way to get it out.

I will ultimately look for a number that is transcendental (starting with Pi), the quickest computations to billions of digits, and a way to incorporate positions with self encoding, and self verifications.

Sebastian

unread,
Feb 4, 2012, 4:41:23 AM2/4/12
to
On 4 Feb., 07:13, Christopher Smith wrote:
> Doing the impossible is interesting

Doing the impossible is impossible by definition. That's what
"impossible" means.

> which is why I try :)

Then, you must be either insane or questioning the impossibility
status of the "thing" you are trying to do.

> Its been proven impossible only as of now.

If it has been proven impossible then it is impossible. That's what
"prove" means.

> If it were to be done, [...]

it would imply that the proof is not actually a proof (i.e. it
contains errors nobody noticed before).

> I believe all
> information exists in Pi, or any transcendental number,

Well, all possible finite digit sequences exist somewhere in the
sequence of PI digits. That's true so far.

> and I am looking for a better way to get it out.

Sounds like you're not interested in checking whether the thing you
are looking for can even exist. It's like you're trying to find a
needle in a huge haystack without knowing for sure whether there is
even a needle in there to be found. Depending on the haystack, it
might be possible by the wonders of mathematics to prove that there
cannot be a needle in it. I'd rather spend some extra hours to check
such a proof for its validity than waste years of my life looking for
something that does not even exist. How about you?

Christopher Smith

unread,
Feb 4, 2012, 5:20:50 AM2/4/12
to
because there might be other things in the haystack worth finding :)

Ernst

unread,
Feb 4, 2012, 1:09:06 PM2/4/12
to
On Feb 3, 10:13 pm, Christopher Smith <madga...@gmail.com> wrote:
> Doing the impossible is interesting, which is why I try :) Its been proven impossible only as of now. If it were to be done, humans would be on another conquest to understand more about why rules can be broken. I believe all information exists in Pi, or any transcendental number, and I am looking for a better way to get it out.
>
> I will ultimately look for a number that is transcendental (starting with Pi), the quickest computations to billions of digits, and a way to incorporate positions with self encoding, and self verifications.

I don't know much about Cantor the mathematician but watching some of
the youtube videos Cantor is a case where we all can take note.
If he can see infinities where others could not I say it's okay to
entertain the impossible.

I have a feeling Transcendental Numbers will be a part of a future
project here. I have an idea of replacing the method of how we
exchange data like we do with Bit Torrent. It is very doable and
would stop the SOPA and other nonsense or at least change the game
significantly.

Hey do you mind if I add you to my friends list? I want to look into
generating Transcendentals in the future and you sound like just the
dreamer to bounce ideas off of.

With the rules can be broken I would change that to Man's Perceptions
can be changed. Scientific Peer types live with rigidity rules that
serve us well but dreamers get soft fluffy pillows for their head if
you get the joke.

If I may give to you I would say consider that all information already
exists in that it is possible for a "thing" to be such as
Transcendentals and that Man is the thing changing in space-time.
What you may find with your imaginings is some underlining principle
or some "Cantor style insights" so more power to you.

By the way I am a common man. I refer to myself as a Friend of
Science not a Peer of Science so I too get the fluffy pillow rather
than the ridged one.
The argument the Peer types make is you are wasting your time not
doing it the Peer-way and there is some truth to that. But I find it
far better to move towards the rigidify of Peer from the fluffy pillow
of dreaming when I create new programs.

The Buddha is credited with saying we must find the middle way. I
like that. I think it may be an idea for you to consider.
Check out the Taoist stuff too for more imagining energy.

Ernst

Ernst

unread,
Feb 4, 2012, 1:13:44 PM2/4/12
to
On Feb 4, 2:20 am, Christopher Smith <madga...@gmail.com> wrote:
> because there might be other things in the haystack worth finding :)

Don't mind the Thought Nazi.

This is called sport fucking and you are the girl to him. In a way
he means well but it's a bit perverse that it is the same tone year
after year and that it becomes a sport fuck party if they get your
goat.

So you can freely read him and he is experienced and knowledgeable
but remember he cannot walk any path he does not know.

Dreaming is okay.. Being a Friend of Science is okay as well as
being a Peer of.
"The Middle way you will find," I say in my best Yoda voice.

Ernst

unread,
Feb 4, 2012, 4:05:15 PM2/4/12
to
Chris, Drop me an email.

I have an interesting idea to bounce off of you.

The email listed is valid. Make the subject line "Transcendental" and
I will capture you for a contact.

I have a lot of emails daily and that will help me find you in the
Stream.

Earl_Colby_Pottinger

unread,
Feb 4, 2012, 11:36:48 PM2/4/12
to
Ernst, you talk big about working outside the box, but so far I have
seen no worthwhile results from you following your big claims.

When are you going to prove that you are right by following up on your
own claims instead?

Thomas Richter

unread,
Feb 6, 2012, 11:27:33 PM2/6/12
to
On 03.02.2012 11:30, Christopher Smith wrote:
> Hello all,
>
> I promised myself I wouldn't write in here unless I absolutely thought I had a decent idea, and I now find myself needing some help. Theres alot here, so please follow along carefully, and if there is a better way I can explain something I will edit the post.

What makes you so certain that this actually compresses anything? Or to
be specific, why do you believe that by juggling numbers as you do you
reach some useful result? If you believe it does, please explain *why*
you think such choices are good ones.

Or even different, would it do any better or worse if I would add the
weekday of the date I'm executing the program and subtract the second
digit of the compile time of the program expressed in milliseconds?



George Johnson

unread,
Feb 7, 2012, 7:24:53 AM2/7/12
to
"Sebastian" <s.ges...@gmail.com> wrote in message
news:49ea0858-4e31-4360...@v14g2000vbc.googlegroups.com...
Actually, do you know what would be a good project to undertake?
Considering that Data Compression is a field of "Knowledge Mining" and
the many tunnels of mathematical failure and discovery have been hollowed
out many times in repetition (oftentimes for the good of the world in the
same way that the personal tape recorder ruined many an aspiring singer's
future career options). Given that so much time and energy is wasted going
down either obviously wrong paths or just rehashing the same old conceptual
dead ends.

I suggest:
Write a book (for longer social value) plus online documents called,
"Data Compression DEAD ENDS".

Kind of an update to the Data Compression FAQ for the casual crowd.
ftp://rtfm.mit.edu/pub/usenet-by-hierarchy/comp/compression/

I figure you either just cover the easy stuff right off in Chapter 1
(all of the data compression ideas that fail instantly when the Escape
Characters which divide Data Set Numbers from Data Set Code) with gentle
humorous commentary on why these fail. Add in why you need to consider the
size cost of each code statement and the first chapter intro is a mix of
invitation and gentle humorous disclamation. Let them know why it is
important to check your work then understand that sometimes the best
knowledge can come from trial & error & correction even if it does not lead
to sellable product.

Chapter 2 would be why pure mathematical data compression fails (from a
Binary math perspective then a Decimal Base 10 dissection). Why you cannot
simply not convert a huge number into base primes without considering the
Escape Character cost and the obvious of Binary 100 * 100 = 10000 (a 3-digit
binary number multiplied by another 3-digit binary number equals a 5-digit
binary number). Ergo, a number converted into prime factors has an equal or
greater number of binary digits as the original number. Base 10 (2*3*7*13 =
546). Base 2 (10*11*111*1101 = 1000100010) (11 binary digits + 4 escape
characters > 10 binary digits). You can see that it actually is cheaper
having the larger number than the small number bunch.

Chapter 3 would be the value of lossly compression and the problems of
having code that compresses, but never decompresses because the math &
concept is fundamentally flawed.

Chapter 4 would be about "Magic Function Theory", the promise and
obvious flaws and why some nut jobs imagine that the 1:1 mapping is somehow
the magical wet blanket for far-too-eager data compression ideas while
ignoring the obvious logical problem of denying that if folks didn't ignore
that the claims of "Magic Function Theory" that no Data Compression would
ever exist.

Chapter 5 can cover just what people have done already, how they failed,
and the good things that came from those efforts (Fractal Compression
becoming the basis for a number of new image processing techniques. Just
leave out a few encouragements after discouraging the folks out there from
rehashing the obvious.

Chapter 6 covers why data set rearrangement in general (shuffling the
card deck) is not profitable for data compression in general (the indexing
cost of shuffling usually costs more than the potential data set future size
reduction). Then move onto Burrows-Wheeler Transform and explaining why it
works (in terms of indexing costs).

I'm not really here to outline a book for you in the entirety, but you
can see the need for creating road markers in the historical records so that
so much precious lifetime is not squandered going down a long Data
Compression road littered on the sides with dead conceptual horses beaten to
a fine mushy paste. To some degrees the book writes itself once you get
going and only requires witty gentle comedy additions and massive editing.
We don't want to discourage the newbies, just gently poke them in the ribs
and say, "Uh nope, go ahead an do this if you need the practice, but this
path you've chosen leads to no practical rewards so us older folks will just
sit on the sidelines and laugh at you if you try to sell it."





Ernst

unread,
Feb 8, 2012, 12:52:07 PM2/8/12
to
On Feb 4, 8:36 pm, Earl_Colby_Pottinger
Earl can you function without attacking me?

I have never seen that either.

Earl my tone and manor is calm. My memory for the new machine arrives
today and soon I can go back to work.
Wish me well on configuring a new computer system.


Ernst

Ernst

unread,
Feb 8, 2012, 12:55:48 PM2/8/12
to
On Feb 7, 4:24 am, "George Johnson" <matri...@charter.net> wrote:
> "Sebastian" <s.gesem...@gmail.com> wrote in message
>     Kind of an update to the Data Compression FAQ for the casual crowd.ftp://rtfm.mit.edu/pub/usenet-by-hierarchy/comp/compression/
Don't forget the final chapter on "We don't know everything."

Ernst

unread,
Feb 8, 2012, 12:53:38 PM2/8/12
to
Tom he is actually interested in Transcendental numbers and
"information" so it's not appropriate. I am interested in them too.

Ernst

Christopher Smith

unread,
Feb 8, 2012, 8:10:43 PM2/8/12
to
> Tom he is actually interested in Transcendental numbers and
> "information" so it's not appropriate. I am interested in them too.
>
> Ernst

Take for example, an oddity with the information of PI, I never looked
for it, but noticed it today.

Take any number, 000-999. I can get those 3 digits down to 10 bits.

Then Take position # 999,999,999,999 in PI. It would take 40 bits to
represent that position in PI.

Take the 9 digits in PI at that position. Lets say 483,765,811. It
would take 30 bits to represent it.

Does the sequence 483,765,811 exist in the first (999,999,999,999+9)
digits of pi, at least 1024 times.

Now does this scale? Is there a minimum number of times a sequence
exists within a certain range of values??

Does a sequence thats 30 digits long appear in ((10^40)+30) digits of
PI at least 1024 times?





Christopher Smith

unread,
Feb 8, 2012, 8:19:40 PM2/8/12
to
Let me refine,

Position 999,999,999,999 = 40 bits

9 digit sequence at that position could be 832,564,322 = 30 bits

2^9 bits=512 position markers

832,564,322 exists in first (999,999,999,999+9) digits of PI > 512
times?

should that scale predictably?

George Johnson

unread,
Feb 8, 2012, 9:15:26 PM2/8/12
to
"Christopher Smith" <madg...@gmail.com> wrote in message
news:622db39b-1735-4013...@s8g2000pbj.googlegroups.com...
http://en.wikipedia.org/wiki/Pi
Practically, one needs only 39 digits of ? to make a circle the size of the
observable universe accurate to the size of a hydrogen atom.

There are binary versions of Pi.
A rather elegant program was written to predict every digit of Pi in
binary by the way.
http://en.wikipedia.org/wiki/Pi
http://en.wikipedia.org/wiki/Bailey%E2%80%93Borwein%E2%80%93Plouffe_formula
The Bailey-Borwein-Plouffe formula (BBP formula) provides a spigot algorithm
for the computation of the nth binary digit of ?. This summation formula was
discovered in 1995 by Simon Plouffe. The formula is named after the authors
of the paper in which the formula was published, David H. Bailey, Peter
Borwein, and Simon Plouffe. Before that paper, it had been published by
Plouffe on his own site.
Since this discovery, many formulas for other irrational constants have been
discovered of the general form.

http://www.befria.nu/elias/pi/binpi.html
11.

00100100 00111111 01101010 10001000 10000101 10100011 00001000 11010011
00010011 00011001 10001010 00101110 00000011 01110000 01110011 01000100
10100100 00001001 00111000 00100010 00101001 10011111 00110001 11010000
00001000 00101110 11111010 10011000 11101100 01001110 01101100 10001001

<256>

01000101 00101000 00100001 11100110 00111000 11010000 00010011 01110111
10111110 01010100 01100110 11001111 00110100 11101001 00001100 01101100
11000000 10101100 00101001 10110111 11001001 01111100 01010000 11011101
00111111 10000100 11010101 10110101 10110101 01000111 00001001 00010111

<512>

10010010 00010110 11010101 11011001 10001001 01111001 11111011 00011011
11010001 00110001 00001011 10100110 10011000 11011111 10110101 10101100
00101111 11111101 01110010 11011011 11010000 00011010 11011111 10110111
10111000 11100001 10101111 11101101 01101010 00100110 01111110 10010110

<768>

10111010 01111100 10010000 01000101 11110001 00101100 01111111 10011001
00100100 10100001 10011001 01000111 10110011 10010001 01101100 11110111
00001000 00000001 11110010 11100010 10000101 10001110 11111100 00010110
01100011 01101001 00100000 11011000 01110001 01010111 01001110 01101001


Christopher Smith

unread,
Feb 8, 2012, 9:18:03 PM2/8/12
to
On Feb 8, 8:15 pm, "George Johnson" <matri...@charter.net> wrote:
> "Christopher Smith" <madga...@gmail.com> wrote in message
>
> news:622db39b-1735-4013...@s8g2000pbj.googlegroups.com...
>
> > Let me refine,
>
> > Position 999,999,999,999 = 40 bits
>
> > 9 digit sequence at that position could be 832,564,322 = 30 bits
>
> > 2^9 bits=512 position markers
>
> > 832,564,322 exists in first (999,999,999,999+9) digits of PI > 512
> > times?
>
> > should that scale predictably?
>
> http://en.wikipedia.org/wiki/Pi
> Practically, one needs only 39 digits of ? to make a circle the size of the
> observable universe accurate to the size of a hydrogen atom.
>
>     There are binary versions of Pi.
>     A rather elegant program was written to predict every digit of Pi in
> binary by the way.http://en.wikipedia.org/wiki/Pihttp://en.wikipedia.org/wiki/Bailey%E2%80%93Borwein%E2%80%93Plouffe_f...
Computation of the digits in billions takes a while..

George Johnson

unread,
Feb 8, 2012, 9:56:04 PM2/8/12
to
"Christopher Smith" <madg...@gmail.com> wrote in message
news:622db39b-1735-4013...@s8g2000pbj.googlegroups.com...
Ideally only one sequence of any number should appear once, any more
than that and your results are sub-optimal. You need to map big numbers to
small and small numbers to even smaller numbers. Optimal 1:1 remap fails
once you can no longer map big numbers to small numbers.

Well, let us review the data set (using location format increments of
0,1,2,3,...)
Binary Pi (under the decimal point).
0 - Found at location 0
1 - Found at location 2
1:1 remapping = not optimal.

Binary Pi (under the decimal point).
(Incrementing location by 1 - Using Small-Endian - going to the
location, reading from righthand-to-left)
00 - Found at location 0
01 - Found at location 2
10 - Found at location 3
11 - Found at location 12
2:1 remapping = not optimal

(Incrementing location by 1 - Using Big-Endian - going to the location,
reading from lefthand-to-right)
00 - Found at location 0
01 - Found at location 1
10 - Found at location 3
11 - Found at location 10
Not optimal

(Incrementing by 2 - Little-Endian)
00 - Found at location 0
01 - Found at location 1
10 - Found at location 2
11 - Found at location 5
Not optimal

(Incrementing by 2 - Big-Endian)
00 - Found at location 0
01 - Found at location 2
10 - Found at location 1
11 - Found at location 5
Not optimal

George Johnson

unread,
Feb 8, 2012, 10:07:30 PM2/8/12
to
"Christopher Smith" <madg...@gmail.com> wrote in message
news:622db39b-1735-4013...@s8g2000pbj.googlegroups.com...
http://en.wikipedia.org/wiki/The_Song_That_Never_Ends
This is a song that never ends,
It just goes on and on my friends,
Some people started singing it not knowing what it was
And they'll continue singing it forever just because,

This is a song that never ends,
It just goes on and on my friends,
Some people started singing it not knowing what it was
And they'll continue singing it forever just because,

This is a song that never ends,
It just goes on and on my friends,
Some people started singing it not knowing what it was
And they'll continue singing it forever just because,

This is a song that never ends,
It just goes on and on my friends,
Some people started singing it not knowing what it was
And they'll continue singing it forever just because...

http://en.wikipedia.org/wiki/How_to_keep_an_idiot_busy_for_hours


Ernst

unread,
Feb 9, 2012, 12:33:01 PM2/9/12
to
On Feb 8, 6:56 pm, "George Johnson" <matri...@charter.net> wrote:


George, I haven't had my cup of coffee yet but I am wide awake after
reading your incredible sharing of information.

Why that saves both myself and Christopher a lot of pondering.

Thank You very much.

I am saving your posts for further reading. I must return to my
previous work but Pi and other irrationals have a place in a future
project simply as what they are.

Ernst

Christopher Smith

unread,
Feb 9, 2012, 3:31:38 PM2/9/12
to
upper bound for existence of information:
(n)=any number with string length divisible by 3. May include leading
zeros.
(Sl)= string length

**(n) will exist in at most (10^(n+3)) digits of p.

Christopher Smith

unread,
Feb 9, 2012, 4:37:03 PM2/9/12
to
> (Sl)= string length of (n)
>
Err

**(n) will exist in at most (10^(Sl+3)) digits of p.

Sebastian

unread,
Feb 10, 2012, 5:41:07 AM2/10/12
to
On 9 Feb., 22:37, Christopher Smith wrote:
> On Feb 9, 3:31 pm, Christopher Smith wrote:
>
> > upper bound for existence of information:

Don't say "information" when you mean "data".

> > (n)=any number with string length divisible by 3. May include leading
> > zeros.
> > (Sl)= string length of (n)
>
> Err
>
> **(n) will exist in at most (10^(Sl+3)) digits of p.

I don't think that's true. Anyhow, what's that got to do with data
compression? Are you looking for a way to encode the position of a "to-
be-compressed" digit sequence in the digits of Π? What makes you think
that this is not a dead end? Where do you see a loophole in the
arguments of the comp.compression FAQ entry about the "magic function
theory"?

Cheers!
SG
0 new messages