Compression factor

[Henrik]

Is there a way to measure the compression factor of a table/column
family? It would be nice to know which compression format is most
effective at compressing your particular data.

Google's BMdiff is, if I understand correctly, running diff against
different columns in the column family, and adjacent rows within the
column family (the call the previous rows/columns dictionaries for the
diff). While bmz runs the diff on the key/values that are in the same
compression block. Are these two about equivalent, do you think, in
effectiveness?

Also, do you prefix compress the keys? As I see it, most keys will
differ in only a few bytes in most cases, so prefix compression is
really helpful here.

[Doug Judd]

The best way to measure the compression ratio is to insert data into Hypertable and then inspect the CellStore files by dumping the trailer, which contains the compression ratio, defined as: 

  compressed_size / uncompressed_size

Here's example output:

$ /opt/hypertable/current/bin/ht csdump /hypertable/tables/query-log/default/AB2A0D28DE6B77FFDD6C72AF/cs2

BLOCK INDEX:

0: offset=0 size=20585 row=5586315

1: offset=20585 size=67742 row=5589539

[...]

945: offset=62186001 size=63272 row=999704

946: offset=62249273 size=39744 row=9999967

sizeof(OffsetT) = 4

BLOOM FILTER SIZE: 0

TRAILER:

[CellStoreTrailerV2]

  fix_index_offset: 62289017

  var_index_offset: 62292831

  filter_offset: 62305781

  index_entries: 0

  total_entries: 5122831

  filter_length: 1373625

  filter_items_estimate: 143309

  filter_items_actual: 5121851

  blocksize: 65536

  revision: 1270524885697818309

  timestamp_min: 1270524813360446944

  timestamp_max: 1270524885697818309

  create_time: 1270524896305355000

  table_id: 1

  table_generation: 1

  flags=0

  compression_ratio: 0.327462

  compression_type: 3

  bloom_filter_mode=ROWS

  bloom_filter_hash_count=6

  version: 2

As far as the BMZ compressor goes, it is specified to AccessGroup granularity.  It works best when the data contains lots of replicated data that is the length of the bmz fingerprint.  The default fingerprint length is 19 bytes.  My guess is that mixing disparate column families within an access group probably won't impact the compression ratio that much, but it is best to group data within a bmz-compressed access group that has lots of fingerprint-length replication.

We currently don't have any prefix compression, but that will be added soon (e.g. within the next couple of months).

- Doug

--
You received this message because you are subscribed to the Google Groups "Hypertable User" group.
To post to this group, send email to hyperta...@googlegroups.com.
To unsubscribe from this group, send email to hypertable-us...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/hypertable-user?hl=en.