Serialized Object as a value in Key-Value store.

[Suraj]

Hi,

We are planning to use protocol buffer to serialized the data before
inserting it into the db. Then we will be inserting this serialized
string into the db.

We will be storing this on SSD so look up throughput is pretty high.
But since SSD's are costly, to save on the disk cost, I am thinking to
compress the serialized string before inserting it into DB.

Do someone have done the benchmarking of using GZipStream on the
binary serialized string?
Also, can you please give me any example of how to do this? I want to
compress the serialized string. So data is in memory and it is not in
file.

Thanks.

[Marc Gravell]

This will depend on many factors:

- how big is each fragment? Very small fragments of *anything* generally get bigger when compressed
- what is the data? If it contains a lot of text data you might see benefits; however, many typical fragments will get bigger when compressed - it depends entirely on the content

In one of our uses, I cheat: I the size above some nominal lower-bound, I *try* GZipStream; the moment this exceeds the original size I kill it, and send the uncompressed original. I it turns out to be smaller, I store that.

This works well for us as the tier that is processing this data has plenty of spare CPU to speculatively try both options.

Marc

> --
> You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
> To post to this group, send email to prot...@googlegroups.com.
> To unsubscribe from this group, send email to protobuf+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
>

[Suraj]

Hi Marc,

Thanks for the reply.

Ours is basically a text data. The record size varies, it is in the
range from around 300Bytes to 3kB. But most of them will be more than
1k. I serialized one record from the production, and found that, the
3kB object, when serialized, became 728Byte string. Our data is
structured and is kind of hierarchical. CPU is not a limitation in our
case too, so I was thinking that if compressing will be helpful to us
to further reduce the data size, then it will help us to save on SSD
cost.

Our proto is something similar to :
Message test {
Optional string key = 1; [16 Byte].
Message inner1{
Optional int id = 1;
Message inner2{
Optional int inner_id = 1;
Optional string entry = 2; [size varies
a lot..from 16 byte to 100 byte.]
}
repeated inner2 rec = 2;
}
repeated inner1 in = 2;
}

But this will be changing over time and we do plan to push lot of the
more information(hence change in the proto) to this store once the
basic version is rolled out.

Thanks and Regards,
Suraj Narkhede

On Sep 22, 11:40 pm, Marc Gravell <marc.grav...@gmail.com> wrote:
> This will depend on many factors:
>
> - how big is each fragment? Very small fragments of *anything* generally get bigger when compressed
> - what is the data? If it contains a lot of text data you might see benefits; however, many typical fragments will get bigger when compressed - it depends entirely on the content
>
> In one of our uses, I cheat: I the size above some nominal lower-bound, I *try* GZipStream; the moment this exceeds the original size I kill it, and send the uncompressed original. I it turns out to be smaller, I store that.
>
> This works well for us as the tier that is processing this data has plenty of spare CPU to speculatively try both options.
>
> Marc
>

[Benjamin Wright]

Protocol buffers does nothing to reduce the size of strings and string
data trends to compress very well. In my experience you can expect
similar gains as if you had a simple text file (very good)

[Eyal Farago]

according to my experience, not so much I'm afraid...

since PB format is length prefixed, or actually tag+length prefixed, the compression rates are not as high as one might expect.

I migrated from an in-house XML format to a PB based format, the performance gain was amazing (both serialize and un-serialize), however the resulting BLOB had lousy compression ratio compared to the XML format. I'm not absolutely sure about the numbers but I think PB compressed to 50% the original size, while XML compressed to 16% of its original size, all in all the compressed PB weighted roughly twice the compressed XML.

I had to roll back the change, fortunately I came up with a different way to use PB which increased performance while maintaining smaller (compressed) BLOBS.

My understanding of this behavior is that the tag+size that prefix each string or message field simply reduces the compressor ability to find repeating regions, nesting messages one into an other (like the XML approach) makes things even worse as it introduces more 'prefixes' into the BLOB, which reduces the possibility of finding any repetitions in the BLOB (which is basically one of two major methods compressors use). My solution was to arrange the message in a way such that all strings are grouped together in one repeated field (plus some other nasty tricks).

BTW, I am a great fan of PB, and I am pushing its usage in my company, but when it's used as a serialization format (rather than a message passing protocol) sometimes you have to bend things a bit...

Eyal.