Parquet footer benchmarks

[Marcin Krystianc]

Hi everyone,

I'm an author of PalletJack (https://github.com/G-Research/PalletJack), a tool we use at G-Research to enable efficient reading of a projection of row groups or columns from large Parquet files. Naturally, when I saw the recent footer proposals, I became curious about the Modular Footer / SoA approach.

I took Rok's benchmarks notebook and extended it with a PyArrow baseline, a Thrift C++ compiler, and a FlatBuffers angle. The results are available at https://github.com/marcin-krystianc/ParquetFooterPlayground.

First of all, I'd like to say that I'm a huge fan of the SoA approach, as it's very read-efficient. It requires far fewer Thrift protocol control bytes (hence the compactness and speed). It also invites efficiency at the programming level, since it avoids allocating many small objects to represent the parsed data. Another nice property is that it's FlatBuffers-friendly too, as the FlatBuffers validation step for it is very cheap.

Anyway, I just wanted to share my results and say thank you for proposing the SoA design - it looks like a solid foundation for the new footer format.

Best regards,
Marcin