Compression in Pinot

[srijanm...@gmail.com]

Hello!

I wanted to know if I can somehow definitively come up with a compression ratio that pinot will get for avro data to pinot segment conversion given the data type and cardinality of the columns in avro. Is there a formula that can be used or some recommendation for a good way to do it?

Thanks

[jennifah...@gmail.com]

Hi!

Thanks for reaching out. I was wondering what you're planning on using the data for to make sure I provide the information you need. Are you looking for the difference between the unzipped segment size (stored on pinot servers) or the gzipped segment(in storage)? For zipped segments, I took some example segments and avro files we had to measure sizes. It seems that avro is around 1.1 times the size of the zipped segment file. The use case I took doesn't include special Pinot features such as star tree index, no dictionary columns, inverted index. 

Best,

Jennifer