bar compression

[sam]

Hey guys,

Been going through the older threads on databases (KDB, Monet) with
interest. Here's one parallel issue that I'd like to throw out here to
you all - as a developer, I'd like to stuff as many bars as possible
into my RAM in order to minimize the number of times I have to read
from main memory. Thus, it makes sense to 'compress' bar information as
much as possible while still maintaining O(1) access times for any
particular piece of bar data. All too often, I've seen implementations
(esp. in Java/C#) where a Bar is a Date/O/H/L/C/Vol/Qty - occupying 8*7
= 56 bytes + overhead. This seems quite excessive, and it must be
possible to encode this much more efficiently - ideally at around 10-15
bytes (although I don't know if that's mathematically possible given
the amount of information we're trying to stuff in there)... but while
still tyring to keep access time to O(1) - I'm no expert in compression
techniques, but maybe someone here has some experience in this?

Any thoughts?

Regards,
Sam

[Joel Reymont]

I think you need to use delta-encoding, i.e. store the difference
between one bar and the next. Then you could store, say, 1 byte per
value (4 bytes for h-o-l-c) and use an extra byte to encode the sign
(up/down) of each value. You could also use 2 bits per value in that
extra byte to encode the sign _and_ whether the value is 1 or 2 bytes.

Instead of storing the date you could fill in the friday value on
weekends or prior day's value on holidays and store the dates
separately. You could store the start and end dates of your price
block this way. The important thing is to avoid "holes" in your price
data.

--
http://wagerlabs.com/

[sam]

Thanks - I had a similar idea, but delta-encoding sounds much better.
Will profile the lot and post.

Regards,
Sam